The latest scan for websites running TYPO3 resulted in a total of 243.382 websites running TYPO3. 5144 new websites running TYPO3 were discovered and 7683 websites stopped using TYPO3.
Latest updates for t3versions
New major version statistics
TYPO3 CMS Composer installers v4 detection
Added support for detection of composer based TYPO3 websites which use
typo3/cms-composer-installers version 4.
New major version statistics
The latest scan for websites running TYPO3 resulted in a total of 247.552 websites running TYPO3. 5338 new websites running TYPO3 were discovered and 7092 websites stopped using TYPO3.
Since the current used Django version was about to be end of life, I took some time to update the whole system.
- Ubuntu 18.04 to 20.04
- Python 3.7 to 3.8
- Django 2.2 to 3.2 LTS
- Dependencies to latest versions
The update went pretty smooth without any noticeable problems.
New major and patch level statistics
The latest scan for websites running TYPO3 resulted in a total of 251.416 websites running TYPO3. 6823 new websites running TYPO3 were discovered and 7361 websites stopped using TYPO3. Most websites that stopped using TYPO3 were running on old and unsupported versions of TYPO3 (mainly 4.5, 6.2, 7.6 and 8.7).
All websites in the database have been analyzed for the used TYPO3 patch version. For 90,66% of all websites, a patch level version ("exact" or "at least" version) could be determined. In case of 53,33%, the exact patch level version was determined.
As a result, only 16,95% of all known and analyzed websites use a supported and secure TYPO3 version.
In order to discover new TYPO3 websites, lists with millions of domains are processed on a regular basis. This process is performed by the t3version bot, which checks the main page of a website for fingerprints of TYPO3.
In rare cases, IPS/IDS systems may consider t3version scan requests as suspicious and alert system administrators. Therefore the t3versions bot now clearly identifies itself by sending a unique user agent string with information about the bot.
New statistics for Q2 2021
The overall amount of TYPO3 websites in the database increased by 598 websites since the last scan.
TYPO3 usage for version 9.5 and 10.4 is constantly increasing and usage for all other (ELTS or unsupported) versions is decreasing.
The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 258.660.176 domains. I used 4 virtual servers (6 cores, 16 GB RAM) for the scan which took 4.5 days to complete and produced nearly 4 TB of traffic.
New statistics for Q1 2021
The overall amount of TYPO3 websites in the database decreased by 6.787 websites since the last scan. 37% of those websites that stopped using TYPO3 switched to Wordpress.
The usage of TYPO3 4.5 and 6.2 is constantly decreasing and over 4000 websites were updated to TYPO3 10.4. The biggest decrease in TYPO3 usage is in Germany, but this is also the market with the most new TYPO3 websites.
The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 38.320.877 domains.
New statistics for Q4 2020 and trends
The new quarterly statistics now include a new chart to show the percentage amount of composer usage per TYPO3 major version. Great to see, that the composer usage increases for each TYPO3 major version.
Also another milestone of the "Real-time TYPO3 Market insights" project has been completed by adding a page which shows several charts with TYPO3 trends. Here the data from the last 3 statistics are used to e.g. visualize how TYPO3 usage develops per country.
The scan engine has also been optimized again to respect several website/webserver misconfiguration issues (e.g. wrong non-SSL to SSL redirects or wrong TYPO3 site configurations)
The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 175.023.211 domains.
Added new domain crawler compoment
A new domain crawler component has been added which makes use of multiprocessing and multithreading features in python. This dramatically increased the speed to crawl huge domain lists for TYPO3 usage.
Each worker (6 CPUs, 16 GB RAM) running the crawler docker container can process ~156 domains/second in average. This results in an average traffic per worker of ~220 GB/day.
New statistics for Q3 2020
The new quarterly statistics now also contains an overview of common security headers that websites use. The most used security header is the 'x-content-type-options' header, which is used in more thanb 35% of all TYPO3 websites in the database
The results are based on a re-scan of all known TYPO3 websites in the database and the scan of ~25 million domains (mainly European region).
Added "t3version" header evaluation
I have noticed, that some TYPO3 website owners block the IP address of the t3versions server to prevent the major version from being shown public. I fully understand the motivation behind this action, but blocking IPs will only work temporary, since a full scan of all websites use several servers with different IP addresses.
If a website owner want to disable the exposure of the version number, this can now easily be configured in TYPO3 by settings an additional header as shown in the example below:
config.additionalHeaders.90.header = t3versions: hide-major
New statistics for Q2 2020
While optimizing the TYPO3 detection engine I discovered 2 major problems which sadly resulted in distorted result in all previous statistics. The engine did not cover the following scenarios known TYPO3 websites in the database:
- Check discovered, that the website is offline / domain unreachable, but did not remove it from the database.
- Check followed domain redirects but did not remove known website, when target of redirect is not using TYPO3
Both scenarios are now covered and the rescan resulted in a decrease of 51635 TYPO3 websites in the database of which 28545 are offline/unreachable.
New statistics for Q1 2020
I have optimized the scan infrastructure, so more processes are automated. So not it is possible to spin up a various amount of scan engines based on Docker, that all connect to a central Redis crawling queue.
As a result of the optimization, I scanned multiple lists containing 49.511.476 european domains in total and also did a rescan of all TYPO3 websites in the database.
First Patch Level Statistics
All available TYPO3 websites in the database were analyzed for the TYPO3 Patch Level Version in use. For websites, where it is clearly possible to analyze the TYPO3 patch level version, only 49.211 (23.91%) out of 205.789 websites use a secure TYPO3 version.
Re-Check of all TYPO3 websites in the database
I updated the TYPO3 fingerprinting and also added an additional check, which tries to identify the new CMS a website uses when it stopped using TYPO3. It was no surprise to me, that about 33% of all websites that stopped using TYPO3 began to use WordPress. Also good to see, that a lot of websites updated either to TYPO3 8.7 (7437) or 9.5 (5270)
Checked list with 140+ mio domains and ~285000 known TYPO3 sites
Since the last Re-Check in January, I bought a big list with over 140 mio. domains and scanned all of them. This scan did find 10735 additional TYPO3 Websites.
Also MaxServ contributed a list of ~285000 known TYPO3 sites. This check resulted in 45.905 new sites in the database.
Re-Check of all TYPO3 websites in the database
I did a re-check of all TYPO3 websites in the database and added 2 new charts. 2284 new sites using TYPO3 were found since the last statistics as a result of manual domain checks of users using t3versions.
You may notice, that there are some really low versions in the "TYPO3 update target version" charts available. Since I also optimized the TYPO3 version identification process in t3versions, some sites with a previous "N/A" version are now identified with the actual TYPO3 version and therefore saved as e.g. "Updated N/A -> 3.7" in the database.
Public release and updated support chart
I released t3version to the public and updated the "Support vs. unsupported versions" chart to show TYPO3 6.2 as ELTS supported.
Remove duplicates and optimize TYPO3 version check
I found out, that 12.297 domains were double indexed, because those sites were indexed starting with and without "www.". I optimized the TYPO3 version check to respect this scenario in the future and removed all duplicates. The total amount of TYPO3 websites is now 292.629
Check of list with 48.146.633 finished
The check of all 48.146.633 domains is finished. As a result, 304.926 websites was found with TYPO3 as content management system.
Increased amount of worker processes and added additional crawling server
Finished crawling 1 mio. domains from the big domain list.
With the previous configuration of the queue system, about 8 domains could be checked per second on the production server (4 CPU cores, 12 GB RAM). Since the total CPU load of the system was really low, I increased the amount of worker processes of the queue system to 64, which increased the the crawling speed to about 23 domains per second.
I also added an additional crawling server (6 CPU cores, 24 GB RAM), which uses 80 worker processes. The server has access to the main database and can check about 34 domains per second.
Bought a list with 48.146.633 european domains
I bought list with 48.146.633 european domain names. Started crawling the domains for TYPO3 websites.
Bought a list with 481.883 domains using TYPO3
I bought a list with 481.883 domains using TYPO3 from an online service specialized on web technology analysis. As a result, the total amount of websites using TYPO3 increased 183.024. After analyzing the domain list i noticed, that it contained many domains, which just were used as redirects for a main domain (e.g. www.domain.dk -> www.domain.com).
Bought a list with 24.092 domains using TYPO3
I bought a list with 24.092 domains using TYPO3 from an online service specialized on web technology analysis. As a result, the total amount of websites using TYPO3 increased to 18.389
Implemented queue system for website crawling
Implemented a queue system, which handles the crawling queue and works with multiple CPU cores.
Imported a lists with 400 TYPO3 websites
I imported a list with about 400 TYPO3 websites. From the 400 websites, 95% were still using TYPO3
Added basic statistics
Added some basic statistics about TYPO3 version usage
Deployment to server
Added auto deployment and deployed first version to production server
I started developing the first version of t3versions.