News

Since some system components were about to be EOL next year, I updated all major components of the t3versions service to the latest versions

• Ubuntu 20.04 to 22.04 LTS
• Python 3.8 to 3.11
• Django 3.2 to 4.2 LTS
• Redis 5 to 6
• Dependencies to latest versions
• Updated custom docker images

Since I wanted to avoid downtime as much as possible, I decided to deploy the updated t3versions app to a completely fresh Ubuntu 22.04 LTS system. The update took quite some time, because all included components needed manual testing, as a central component for the queue system was replaced.

The latest scan for websites running TYPO3 resulted in a total of 229.268 websites running TYPO3. 8.324 new websites running TYPO3 were discovered and 7.978 websites stopped using TYPO3.

Additionally, a list of 265 127 101 domains have been crawled and a new major version statistics has been generated.

The detection engine has again been optimized to detect TYPO3 websites, which are only served using SSL and where the non-ssl request to the domain resulted in an unexpected server response (e.g. webserver default website or even directory index).

Statistics for 09/2023

With TYPO3 composer v4 and v5 installer TYPO3, the path to public accessible files has been obfuscated with a hash. This makes it harder for external analysis tools like t3versions to fingerprint websites on file basis.

In order to still provide a regular patch level statistics which covers TYPO3 websites using composer v4 and v5 installer, the fingerprinting analysis has been extended to support the obfuscation of public accessible files.

In order to avoid my worker servers getting blocked, I changed the request handling in t3versions, so GET requests to remote servers can use proxy servers. When scanning for TYPO3 websites and when analyzing known TYPO3 websites for version changes, my worker servers will now use random proxy servers from a pool of private squid instances

Additionally, a list of 262 365 833 domains have been crawled and a new major version statistics has been generated.

Statistics for Q1 2023

The TYPO3 detection routine has been updated and is now able to detect TYPO3 12.4 LTS.

Additionally, the TYPO3 detection routine has now been extended to verify, if a t3versions analysis request has been blocked by a WAF. If the TYPO3 detection routine detects a WAF, a known TYPO3 website will temporarily not be removed from the database. Instead, an extended (still under development) analysis routine will be used to check the website using proxied GET request.

The latest scan for websites running TYPO3 resulted in a total of 236.669 websites running TYPO3. 12.555 new websites running TYPO3 were discovered and 6.779 websites stopped using TYPO3.

Originally, over 20.000 new TYPO3 website hosted in Hongkong were detected. Those where however false positives, since those websites used "TYPO3 CMS" as meta tag without actually running TYPO3.

A not so new but quite annoying problem is the fact, that the IP adresses of my scan servers get added to IP reputation databases resulting in some webservers blocking TYPO3 scan requests. I will find a solution on how to deal with this scenario.

Statistics for Q4 2022

The initial TYPO3 detection routine checks the rendered HTML of a website for TYPO3 fingerprints. If a website did embed TYPO3 resources (e.g. external images or javascript), the website could have been identified as TYPO3 website, if there were more than 3 TYPO3 fingerprints on the website.

The initial TYPO3 detection routine has now been hardened to reduce the amount of false positives.

The latest scan for websites running TYPO3 resulted in a total of 243.382 websites running TYPO3. 5144 new websites running TYPO3 were discovered and 7683 websites stopped using TYPO3.

Statistics for Q2 2022

Added support for detection of composer based TYPO3 websites which use typo3/cms-composer-installers version 4.

The latest scan for websites running TYPO3 resulted in a total of 247.552 websites running TYPO3. 5338 new websites running TYPO3 were discovered and 7092 websites stopped using TYPO3.

Statistics for Q1 2022

Since the current used Django version was about to be end of life, I took some time to update the whole system.

• Ubuntu 18.04 to 20.04
• Python 3.7 to 3.8
• Django 2.2 to 3.2 LTS
• Dependencies to latest versions

The update went pretty smooth without any noticeable problems.

The latest scan for websites running TYPO3 resulted in a total of 251.416 websites running TYPO3. 6823 new websites running TYPO3 were discovered and 7361 websites stopped using TYPO3. Most websites that stopped using TYPO3 were running on old and unsupported versions of TYPO3 (mainly 4.5, 6.2, 7.6 and 8.7).

Statistics for Q4 2021

All websites in the database have been analyzed for the used TYPO3 patch version. For 90,66% of all websites, a patch level version ("exact" or "at least" version) could be determined. In case of 53,33%, the exact patch level version was determined.

As a result, only 16,95% of all known and analyzed websites use a supported and secure TYPO3 version.

TYPO3 Patch Version Statistics Jan. 9, 2022

In order to discover new TYPO3 websites, lists with millions of domains are processed on a regular basis. This process is performed by the t3version bot, which checks the main page of a website for fingerprints of TYPO3.

In rare cases, IPS/IDS systems may consider t3version scan requests as suspicious and alert system administrators. Therefore the t3versions bot now clearly identifies itself by sending a unique user agent string with information about the bot.

More information about t3versions bot

The overall amount of TYPO3 websites in the database increased by 598 websites since the last scan.

TYPO3 usage for version 9.5 and 10.4 is constantly increasing and usage for all other (ELTS or unsupported) versions is decreasing.

The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 258.660.176 domains. I used 4 virtual servers (6 cores, 16 GB RAM) for the scan which took 4.5 days to complete and produced nearly 4 TB of traffic.

Statistics for Q2 2021

The overall amount of TYPO3 websites in the database decreased by 6.787 websites since the last scan. 37% of those websites that stopped using TYPO3 switched to Wordpress.

The usage of TYPO3 4.5 and 6.2 is constantly decreasing and over 4000 websites were updated to TYPO3 10.4. The biggest decrease in TYPO3 usage is in Germany, but this is also the market with the most new TYPO3 websites.

The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 38.320.877 domains.

Statistics for Q1 2021

The new quarterly statistics now include a new chart to show the percentage amount of composer usage per TYPO3 major version. Great to see, that the composer usage increases for each TYPO3 major version.

Also another milestone of the "Real-time TYPO3 Market insights" project has been completed by adding a page which shows several charts with TYPO3 trends. Here the data from the last 3 statistics are used to e.g. visualize how TYPO3 usage develops per country.

The scan engine has also been optimized again to respect several website/webserver misconfiguration issues (e.g. wrong non-SSL to SSL redirects or wrong TYPO3 site configurations)

The results are based on a re-scan of all known TYPO3 websites in the database and the scan of 175.023.211 domains.

Statistics for Q4 2020

A new domain crawler component has been added which makes use of multiprocessing and multithreading features in python. This dramatically increased the speed to crawl huge domain lists for TYPO3 usage.

Each worker (6 CPUs, 16 GB RAM) running the crawler docker container can process ~156 domains/second in average. This results in an average traffic per worker of ~220 GB/day.

The new quarterly statistics now also contains an overview of common security headers that websites use. The most used security header is the 'x-content-type-options' header, which is used in more thanb 35% of all TYPO3 websites in the database

The results are based on a re-scan of all known TYPO3 websites in the database and the scan of ~25 million domains (mainly European region).

Statistics for Q3 2020

I have noticed, that some TYPO3 website owners block the IP address of the t3versions server to prevent the major version from being shown public. I fully understand the motivation behind this action, but blocking IPs will only work temporary, since a full scan of all websites use several servers with different IP addresses.

If a website owner want to disable the exposure of the version number, this can now easily be configured in TYPO3 by settings an additional header as shown in the example below:

While optimizing the TYPO3 detection engine I discovered 2 major problems which sadly resulted in distorted result in all previous statistics. The engine did not cover the following scenarios known TYPO3 websites in the database:

• Check discovered, that the website is offline / domain unreachable, but did not remove it from the database.
• Check followed domain redirects but did not remove known website, when target of redirect is not using TYPO3

Both scenarios are now covered and the rescan resulted in a decrease of 51635 TYPO3 websites in the database of which 28545 are offline/unreachable.

Statistics for Q2 2020

I have optimized the scan infrastructure, so more processes are automated. So not it is possible to spin up a various amount of scan engines based on Docker, that all connect to a central Redis crawling queue.

As a result of the optimization, I scanned multiple lists containing 49.511.476 european domains in total and also did a rescan of all TYPO3 websites in the database.

Statistics for Q1 2020

All available TYPO3 websites in the database were analyzed for the TYPO3 Patch Level Version in use. For websites, where it is clearly possible to analyze the TYPO3 patch level version, only 49.211 (23.91%) out of 205.789 websites use a secure TYPO3 version.

Patch Level Statistics for 04.12.2019

I updated the TYPO3 fingerprinting and also added an additional check, which tries to identify the new CMS a website uses when it stopped using TYPO3. It was no surprise to me, that about 33% of all websites that stopped using TYPO3 began to use WordPress. Also good to see, that a lot of websites updated either to TYPO3 8.7 (7437) or 9.5 (5270)

Statistics for 16.09.2019

Since the last Re-Check in January, I bought a big list with over 140 mio. domains and scanned all of them. This scan did find 10735 additional TYPO3 Websites.

Also MaxServ contributed a list of ~285000 known TYPO3 sites. This check resulted in 45.905 new sites in the database.

I did a re-check of all TYPO3 websites in the database and added 2 new charts. 2284 new sites using TYPO3 were found since the last statistics as a result of manual domain checks of users using t3versions.

You may notice, that there are some really low versions in the "TYPO3 update target version" charts available. Since I also optimized the TYPO3 version identification process in t3versions, some sites with a previous "N/A" version are now identified with the actual TYPO3 version and therefore saved as e.g. "Updated N/A -> 3.7" in the database.

I released t3version to the public and updated the "Support vs. unsupported versions" chart to show TYPO3 6.2 as ELTS supported.

I found out, that 12.297 domains were double indexed, because those sites were indexed starting with and without "www.". I optimized the TYPO3 version check to respect this scenario in the future and removed all duplicates. The total amount of TYPO3 websites is now 292.629

The check of all 48.146.633 domains is finished. As a result, 304.926 websites was found with TYPO3 as content management system.

Finished crawling 1 mio. domains from the big domain list.

With the previous configuration of the queue system, about 8 domains could be checked per second on the production server (4 CPU cores, 12 GB RAM). Since the total CPU load of the system was really low, I increased the amount of worker processes of the queue system to 64, which increased the the crawling speed to about 23 domains per second.

I also added an additional crawling server (6 CPU cores, 24 GB RAM), which uses 80 worker processes. The server has access to the main database and can check about 34 domains per second.

I bought list with 48.146.633 european domain names. Started crawling the domains for TYPO3 websites.

I bought a list with 481.883 domains using TYPO3 from an online service specialized on web technology analysis. As a result, the total amount of websites using TYPO3 increased 183.024. After analyzing the domain list i noticed, that it contained many domains, which just were used as redirects for a main domain (e.g. www.domain.dk -> www.domain.com).

I bought a list with 24.092 domains using TYPO3 from an online service specialized on web technology analysis. As a result, the total amount of websites using TYPO3 increased to 18.389

Implemented a queue system, which handles the crawling queue and works with multiple CPU cores.

I imported a list with about 400 TYPO3 websites. From the 400 websites, 95% were still using TYPO3

Added some basic statistics about TYPO3 version usage

Added auto deployment and deployed first version to production server

I started developing the first version of t3versions.