Get in touch

We use HubSpot CRM to process and manage contact and information requests. Please accept the "Functional Cookies" and reload the page to load the contact form.

Insights /
Blog /
Tech

Insights / Blog / Tech

The importance of Open Source work using the example of the TYPO3 Crawler

October 01, 2020

Tomas MikkelsenDeveloper

One of the myths about Open Source is that it is unpaid work done in your spare time. This is not always the case: At AOE, we build most of our software solutions based on Open Source, so a big part of our contributions is done during regular working hours. We are continuously improving and adapting the software to meet our customers' needs. Everyone benefits from this approach: the developers, the customers, the Open Source community – and AOE itself. How? Let me show you using the example of the TYPO3 Crawler.

Some facts about the TYPO3 Crawler

The TYPO3 Crawler is licensed under the terms of Open Source. It is one of the oldest extensions in the TYPO3 eco-system, and I think the oldest created and maintained by AOE. By now, it has more than 150,000 downloads and more than 70 versions released. The first one was released on December 22, 2005. Compared to others, TYPO3 Crawler is a “small” TYPO3 extension. It helps you with different tasks in your TYPO3 CMS:

It can be used for filling the Search Index (indexed_search extension).
It can be used as a cache warm-up carried out during deployments.
It can in general be used to make sure your recent edited content is cached for the end user to have a faster website.

Since releasing the first version, we at AOE have continuously been working with the TYPO3 Crawler in projects such as Sony Playstation Network, and it is still used in the congstar project to the present day.

Download via packagist.org or extensions.typo3.org.

The upgrade: TYPO3 9 LTS and 10 LTS (Long-Term Support)

On November 11, 2018, the TYPO3 9 LTS was added as a requirement for the TYPO3 Crawler extension. The goal: making the Crawler compatible with TYPO3 9 LTS. However, developments in the TYPO3 core have passed the continuing development of the Crawler as the TYPO3 10 LTS was released. Therefore, it was decided to expand the original goal of TYPO3 9 LTS compatibility. The result: The TYPO3 Crawler now not only supports TYPO3 9, but also TYPO3 10 LTS.

The following key changes and improvements were made:

GuzzleHttp as HTTP Client
symfony/console for CommandController for CLI and Scheduler Tasks
Backend Module uses Fluid for templating
Rector to CI Pipeline
Implemented Doctrine for Database handling
switched to Site Handling instead of sys_domain
dropped support for TYPO3 8 & TYPO3 7
dropped support for PHP <7.2

Some fun facts

Since the beginning of work in November 2018, there have been

~ 170 commits
13 contributors
319 files changed, 8236 insertions(+), 22502 deletions(-)

Benefits of working with Open Source in context of the TYPO3 Crawler

For the developers and the Open Source community:

Additional source of learning: Working with Open Source gives the developers a peek into a lot of different software, code stylings, ways to solve everything from complex to simple problems.
Getting help from the outside if needed: As seen in the fun facts, the work was not done by AOE alone. There were contributions from the outside as well.
Motivation: Most Open Source communities are very open and have a welcoming attitude towards new joiners. Collaboration within these communities can be very motivating and beneficial for a developer.
Faster problem solving: As there are many developers with different experience, approaches, and perspectives, bugs can be fixed much faster.
Interaction: What we did not expect to this extent when updating the extension was the increased user interaction within the community. The amount of feedback, issues, new ideas, etc. have grown a lot after the 9.0.0 release. Because of this, the update enabled us to develop a better extension with added value for the users.

For our clients:

Cost efficiency: License cost is always an argument when talking about Open Source. Compared to a commercial CMS system, TYPO3 CMS costs 0€, which means that the spared license costs can be used for development of custom features not supported by the system yet.
Benefits from collaborative work: Faster bug fixing, maintenance by the community and the constant development of new features are aspects that the clients benefit from as well.
Independence: Being more up-to-date offers the possibility to try out, use, and learn new technologies, which pays into the work done for AOE’s clients. They will not depend on old libraries and components, newer and more modern components make your code easier to maintain, and move some of the complexity to other components that you require instead of developing them yourself.

For AOE:

Improved overall skills: As mentioned earlier, a developer has an additional learning source when working with Open Source. This results in an improved skill set, as one becomes more familiar with new technologies, best practices, and so on. These are all improvements that reflect in client work done at AOE.
Relationship with clients: The relationship with our clients benefits enormously when our behavior as a service provider demonstrates that we always have their best outcome in mind and do not want to just sell them costly technology. Working with Open Source and therefore reducing the clients’ spend gives us more scope to use budgets for other services such as the development of customized features.

This shows that Open Source work not only helps developers to refine their skills and find better approaches to complex problems, but also makes AOE as a company more competitive: We are in touch with a lot of developers, we learn to view problems from different perspectives and we constantly become acquainted with new technologies and approaches. All of that directly benefits all parties involved: developers, community, clients and AOE itself.

The importance of Open Source work using the example of the TYPO3 Crawler

Some facts about the TYPO3 Crawler

The upgrade: TYPO3 9 LTS and 10 LTS (Long-Term Support)

Some fun facts

Benefits of working with Open Source in context of the TYPO3 Crawler

For the developers and the Open Source community:

For our clients:

For AOE:

Related links:

Go, Development Frameworks & Flamingo #3: Introduction to Flamingo

Go, Development Frameworks & Flamingo #2: Pros and Cons of Development Frameworks

Go, Development Frameworks & Flamingo #1: The Language Choice: JUST GO FOR IT!