v0.5

README.md

@@ -1,79 +1,80 @@
-Typo3-Enumerator
+# Typo3Scan
 ===============
 
-Typo3-Enumerator is an open source penetration testing tool that automates the process of detecting the [Typo3](https://typo3.org) CMS and it's installed [extensions](https://typo3.org/extensions/repository/?id=23&L=0&q=&tx_solr[filter][outdated]=outdated%3AshowOutdated) (also the outdated ones).
+Typo3Scan is an open source penetration testing tool that I wrote to automate the process of detecting the [Typo3](https://typo3.org) CMS and it's installed [extensions](https://extensions.typo3.org/).
-If the --top parameter is set to a value, only the specified most downloaded extensions are tested.
+It also has a database with known vulnerabilities for core and extensions. 
 
-It is possible to do all requests through the [TOR Hidden Service](https://www.torproject.org/) network.
+Typo3Scan does not exploit any vulnerabilities! It´s soley purpose was to enumerate version info and installed extensions in penetration tests ever since.
 
-Installation
+**Note:** 
+When I started this project many years ago, the version information could be easily read from text files (Readmes, Changelogs, etc.). Since then a lot has changed.
+Typo3 now restricts access to directories and files by default and since the use of [Composer](https://github.com/composer/composer), version information of extensions are not available in files anymore.
+In addition, various basic functions have changed over time.
+For these reasons this tool will probably *not receive further major releases*.
+
+
+## Installation
 ----
 
-You can download the latest tarball by clicking [here](https://github.com/whoot/Typo-Enumerator/tarball/master) or latest zipball by clicking  [here](https://github.com/whoot/Typo-Enumerator/zipball/master).
+You can download the latest tarball by clicking [here](https://github.com/whoot/Typo3Scan/tarball/master) or latest zipball by clicking  [here](https://github.com/whoot/Typo3Scan/zipball/master).
 
-Preferably, you can download Type-Enumerator by cloning the [Git](https://github.com/whoot/Typo-Enumerator) repository:
+Preferably, you can download Type3Scan by cloning the [Git](https://github.com/whoot/Typo3Scan) repository:
 
-    git clone https://github.com/whoot/Typo-Enumerator.git
+    git clone https://github.com/whoot/Typo3Scan.git
 
-Typo-Enumerator works with [Python](http://www.python.org/download/) version **3.x** on Debian/Ubuntu, RedHat and Windows platforms.
+Typo3Scan works with [Python 3](http://www.python.org/download/) version **3.7** on Debian/Ubuntu and Windows platforms.
 
-You might need to install following packages:
+You can install all required packages with pip3:
 
-* [Requests](https://pypi.python.org/pypi/requests/)
+	pip install -r requirements.txt
-* [Colorama](https://pypi.python.org/pypi/colorama)
 
-You can install the packages with pip3 on Debian/Ubuntu and Windows:
+## Usage
-
-	pip3 install requests colorama
-
-On Redhat you can install all needed packages with easy_install:
-
-	easy_install argparse
-	easy_install requests
-	easy_install colorama
-
-If you want to use Typo-Enumerator with TOR, you need the [SocksiPy](https://sourceforge.net/projects/socksipy/) module.
-
-Usage
 ----
 
 To get a list of all options use:
 
-    python3 typo3_enumerator.py -h
+    python typo3scan.py -h
 
-You can use Typo3-Enumerator with domains:
+You can use Typo3Scan with domains:
 
-	python3 typo3_enumerator.py -d DOMAIN [DOMAIN ...] [--top VALUE]
+	python typo3scan.py -d DOMAIN [DOMAIN ...] [--vuln]
 
 Or with a file with a list of domains:
 
-	python3 typo3_enumerator.py -f FILE [--top VALUE]
+	python typo3scan.py -f FILE [--vuln]
 
 Example:
-Test if Typo3 and top 200 downloaded extensions are installed on 192.168.0.24:
 
-	python3 typo3_enumerator.py -d 192.168.0.24/testsite --top 200
+	python typo3scan.py -d http://dev001.vm-typo3.loc --vuln
 	
-![ScreenShot](/doc/Screenshot.jpg)
+![core_vulns](/doc/core_vulns.jpg)
+
+![ext_vulns](/doc/ext_vulns.jpg)
 
 
-Bug Reporting
+## Bug Reporting
-----
-Bug reports are welcome! Please report all bugs on the [issue tracker](https://github.com/whoot/Typo-Enumerator/issues).
-
-Links
 ----
 
-* Download: [.tar.gz](https://github.com/whoot/Typo-Enumerator/tarball/master) or [.zip](https://github.com/whoot/Typo-Enumerator/archive/master.zip)
+Bug reports are welcome! Please report all bugs on the [issue tracker](https://github.com/whoot/Typo3Scan/issues).
-* Changelog: [Here](https://github.com/whoot/Typo-Enumerator/blob/master/doc/CHANGELOG.md)
+
-* TODO: [Here](https://github.com/whoot/Typo-Enumerator/blob/master/doc/TODO.md)
+I´m developing this in my spare time. If you like my work, please consider supporting my coffee consume:
-* Issue tracker: [Here](https://github.com/whoot/Typo-Enumerator/issues)
+
+[![Buy me a coffee](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/whoot)
+
+
+## Links
+----
+
+* Download: [.tar.gz](https://github.com/whoot/Typo3Scan/tarball/master) or [.zip](https://github.com/whoot/Typo3Scan/archive/master.zip)
+* Changelog: [Here](https://github.com/whoot/Typo3Scan/blob/master/doc/CHANGELOG.md)
+* Issue tracker: [Here](https://github.com/whoot/Typo3Scan/issues)
 
 # License
+----
 
-Typo3 Enumerator - Automatic Typo3 Enumeration Tool
+Typo3Scan - Automatic Typo3 Enumeration Tool
 
-Copyright (c) 2015-2017 Jan Rude
+Copyright (c) 2015-2020 Jan Rude
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -86,4 +87,4 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 
 You should have received a copy of the GNU General Public License
-along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)

doc/CHANGELOG.md

@@ -1,3 +1,13 @@
+## Version 0.5
+
+* Rename to Typo3Scan
+* Code cleanup
+* Many improvements
+* Using Database to store extensions, version info and vulnerabilities (Core and Extensions)
+* TOR support dropped
+* *--top* parameter support dropped. Can not be used anymore, because download counter is not used anymore.
+  Instead *--vuln* parameter can be used to check for extensions with known vulnerabilities.
+
 ## Version 0.4.5.2
 
 * Fixed error on update
@@ -41,11 +51,11 @@
 * Fixed link to socksipy for python 3
 * Fixed bug in versionsearch
 * Fixed TOR issues
-* Fixed some bugs
+* Bugfixes
 
 ## Version 0.4
 
-* Using Python 3.x now!
+* Using Python 3 now!
 * Using classes and objects
 * Better searching algorithm
 * Better threading (fixed local network DOS when cheking a lot of extensions)
@@ -72,7 +82,7 @@
 ## Version 0.3
 
 * Using modules instead of one class
-* Accepting now strg+c when in multithreaded mode
+* Accepting now strg+c when in multi-threaded mode
 * Update function will now generate a list with extensions instead of an xml. This list is sorted by default (download count). Loading this file is much faster than parsing the xml everytime.
 * It is now possible to use only TOR, Privoxy, or TOR with Privoxy (in order to prevent DNS leakage).
 * Max. threads are set to 10 to prevent connection issues and DoS, default threads are now 7.

doc/TODO.md

@@ -1,6 +0,0 @@
-# TODO
-
-* Newer Typo3 installations use /typo3/index.php?id=xxx 
-* Stop extension enumeration with ctrl-c
-* maybe store extensions and findings in database
-* code cleanup

extensions/experimental_extensions

@@ -1,178 +0,0 @@
-alternative_rte
-kh_photoweb
-rte_mswordclean
-frontendformslib
-mmforeign
-cc_cbrowse
-articles
-sl_css_imgtext
-google_api_search
-formbuilder
-dam_demo
-nicosdirectory
-em2_test
-xml_contentrendering
-xajax_tutor
-csevents
-lz_chess
-convertutf
-beuser_tracking
-mz_miniworkflow
-xajax_example
-pdf_export
-cite
-jp_events
-sk_svg
-ilis_newsticker_doc
-bm_basics
-cwt_hcl
-ahxol
-rsp_webmail
-ameos_demo_formidable
-svo_tvplaintext
-eu_sso_horde
-p2_menudom
-zor_ts2str
-aofeusersystem
-test_contux
-ggt_catwalk_template
-cp_validator
-cc_tsquicklink
-hsr_charts
-hjm_singlesignon
-mpg_staff
-link_by_pageid
-fl_googleadsense
-sici_tmplonfs
-masi_utf8fe36
-jg_wishlist
-ameos_demoformidable050
-kb_test_em
-skraemer_tmplsel
-doc_formbuilder_fr
-openimmoman
-jm_recaptcha_example
-programm_roadmap
-wag_ntlm_sso
-pr_mailcampaign
-rhu_excelexplorer
-felib
-eim2mvc
-cherries
-sm_charsethelper
-mvwa_fortune
-ws_test
-survey
-masi_utf8fs
-rhu_member
-dsn_sopform
-ggt_catwalk_wardrobe
-rg_yfm
-dsn_sopdisplay
-cps_eventsnow
-test_wj
-feusers2xml
-svq_feuser_filemanager
-ch_mandelbrot
-subdoktypes
-dr_directors_list
-ck_heim
-sm_test123
-tvp_clipboard
-kickstarter__mvc_ex
-joytopia
-tvp_newcewizard
-abcstarter
-tw_shop
-vegb2cmnt
-party
-ms_eidtest
-extfileupload
-svq_fe_user_mailform
-efs
-test_uploaddependency
-jhe_dam_extender
-dbreplace
-spriteiconoverview
-eventmanagement
-ms_fluid
-bb_easyforms
-abcconfig
-ajax_report
-smu_chc_ext
-ch_flash_carrousel
-tcaobjects_demo
-jr_webmail
-wsefs
-rhu_csvimport
-pb_rsslaufschrift
-european
-ch_bramacroofsimulator
-p2_langfix_42
-clanbase
-ter_tests
-meta_openoffice
-st_validation_lpl
-rhu_events
-t3info
-sort_table
-ch_bramacproducts
-organizacionacademica
-bonus
-maja_condrequired
-alumnos
-hh_multipageform_example
-lz_lp_dm_log_fe
-dsxsyndication
-zitatdt
-ba_company
-svq_ebay
-automator
-rm_staticfile
-contactformgenerator
-rg_links
-audio_conversion
-error
-mbbrowserid
-wow_raid
-rg_usuarios
-mf_trainmanagement
-rg_patrocinio
-sp_newsteaserbox_hookexample
-redirectlog
-party_tests
-rb_tf_database
-asvtiger
-belink_syslang
-buildtools
-rg_empresas
-rf_library
-treppenpfosten_katalog
-tc_fbconnect
-ffunews
-dre_besearch
-elnews
-ft3_empty
-downloads
-ecs_steam
-boards
-tagger
-dbal_utility
-jh_pwcomments_plugin
-mr_base_config
-femanagerextended
-xdbmysql
-air_table
-ter_upload_test
-og_base
-cabag_deploy
-jh_extstatus
-moox_news_twitter
-reint_mailtask_example
-tgm_kickstart
-visitorlist
-simplemvc_helloworld
-ctefan_test
-ckeditor
-lo_backendhelper
-moox_news_geoinfo

extensions/outdated_extensions

@@ -1,321 +0,0 @@
-newloginbox
-rtehtmlarea
-ter_update_check
-csh_de
-csh_nl
-th_mailformplus
-sr_static_info
-date2cal
-fh_library
-kb_md5fepw
-rte_pb_htmlarea
-maag_randomimage
-kj_recycler
-news_plus
-naw_securedl
-css_styled_imgtext
-csh_fr
-dam_file
-danp_documentdirs
-a1_ttnews
-sr_rtehtmlarea_xpblue
-eco_gal
-dmmjobcontrol
-rte_conf
-ws_sitemap
-smile_workflow
-typo3_tut
-modern_skin
-csh_ru
-fancycorners
-wt_directory
-newloginbox_tmplable
-bzd_staff_directory
-cc_devlog
-sr_language_detect
-rte_chooser
-sp_news_catimgs
-doc_core_tsref
-dkd_newsmulticats
-csh_dk
-kb_unpack
-realurlsettings
-beuser_ip_lock
-csh_es
-addressgroups
-fl_staticfilecache
-cc_metaexif
-sr_rtehtmlarea_bluelook
-jf_headerslide
-doc_core_tsconfig
-ml_links
-imailframe
-kb_cont_slide
-cc_metaexec
-csh_it
-news_pack
-pk_limitmenutolang
-df_dmailer
-doc_core_tstemplates
-pt_payment
-csh_jp
-livestatframe
-fed
-wmdb_contentwizard
-doc_core_inside
-smile_workflowdoc
-csh_pl
-ah_flashinnews
-xtemplate
-mh_auto_alias
-gb_workflow
-ameos_feuser_manager
-tableswithoutp
-doc_core_cgl
-sg_zfeeditlib
-aau_pbsurvey
-joboffers_feedit
-gforms
-kb_shop
-danp_md5fepassword
-doc_core_ts
-cal_tt_news_service
-rrzn_pagelinks
-me_templavoilalayout2
-gb_feuserregistratio
-av_weblinks
-doc_core_api
-goof_fotoboek_fix1
-maag_formcaptcha
-doc_tut_quickstart
-sg_feautologin
-lz_https
-cc_infotablesmgm
-rtp_smarty
-csh_cz
-partner_fe
-patch1822
-csh_ch
-csh_ar
-mw_macmenu
-doc_core_services
-sg_feautologinnest
-csh_kr
-news_displaysingle
-cal_news_event_service
-smile_categorization
-framed_header
-efafontsize
-doc_inst_upgr
-df_feuser_register_ext
-fl_ebayinfo
-sr_readmail_analyse
-googleanalytics
-csh_tr
-csh_hu
-onet_ttprodoffers
-danp_extrdfexport
-df_mailformplus_ext
-csh_se
-sg_fenewsedit
-csh_sk
-cobwebphpadsnew
-dynbeedit
-glossarysearch
-csh_gr
-csh_hk
-csh_br
-dubletfinder
-prototypejs
-wa_contentrenderinghook
-hsapp_longerfeusername
-de_contentorganizer
-danp_skinsupport
-alt_forms_field_title
-danp_webcatalog
-csh_si
-abz_labels
-gp_404page
-csh_bg
-csh_ua
-formidabledatetime
-mh_multimedia_ext
-sav_library
-eco_cal
-stucki_cache_imagesizes
-perfectlightboxjquery
-csh_pt
-gt_typo3_localization
-csh_hr
-csh_ro
-csh_fi
-tmpl_ice_3columns
-csh_no
-mhnotifychanger
-doc_ephp_install_fr
-csh_ca
-kb_renmultp
-cron_autoaccesskeys
-maob_mmcache
-cc_meta_xmp
-csh_th
-wildside_extbase
-extendcobj
-essayeca_cal
-sms_directshortcuts
-tm_dl
-csh_ba
-csh_he
-csh_lt
-salutationswitcher
-csh_is
-sg_adradd
-xds_sermon_base
-csh_gl
-csh_lv
-csh_et
-phpbb3_auth
-doc_l10nguide
-html5meta_t3lib_pagerenderer
-twittersearch
-tex_script
-donations
-sp_betterflex
-localphpinclude
-tm_classes
-fl_langtranslate
-danp_userlisttemplate
-cobweb_protector
-tebay
-rtehtmlarea_definitionlist
-yag_theme_perfectlightbox
-eco_content
-softwarecenter
-csh_vn
-tm_minijoboffers
-paysuite
-idaa_fe_utilies
-mailformplusplus
-go_maps_ap
-ak_mobile_device
-iwbase
-eu_correcturls
-rtehtmlarea_dummyplugin
-ws_imageeffects
-jq_lightbox2
-ad_google_maps_api
-kk_mailformpluslist
-yag_theme_fancybox
-maag_sendmail
-xds_sermon_player
-powermail_static_template
-egovapi
-ts45min_de
-t3blogjquery
-cl_jquery
-googlequery
-extensionlist
-fe_db_browser
-mm_forum_comments
-ab_swiftmailer
-ch_hidedefaultlang
-larsp_fussballde_js
-stfl_startendtime
-completebackup
-speedy
-sav_library_extends
-tinysource
-mm_forum_news
-flow4t3
-browser_tut_ajax_en
-larsp_tagcloud
-yiid_like
-tm_qsdb
-mpm
-templatelib
-doc_guide_install
-cacheexpire
-facebook
-mpr
-displaycontroller_advanced
-smile_form_archive
-tagpackprovider
-dfluess
-doc_core_tca
-redirection
-jhe_adventcalender
-sav_library_example5
-maag_imagerotator
-xliff
-remote_server
-metadata_ts
-doc_tut_ts45
-datadisplay
-form4_doktypes
-st_readmore
-mak_randlistnum
-static_info_tables_ga
-extended_sys_note
-advancedform
-delete_staticfile_by_3party
-ics_errorhandler
-ods_workspace_mail
-tm_gallery
-extend_dcdgooglemap
-ttnews_href_marker
-doc_tut_editors
-st_metatags
-ics_templavoila_mirgation_tool
-doc_guide_security
-doc_core_skinning
-ttnewscacheexpire
-form4_contentpagination
-realurl_autoconf_autodelete
-paymentlib_dibs
-paymentlib_quickpay_dk
-smile_jumpurl_fix
-tgm_gallery
-tsincludeorder
-tm_cssfilelinks
-tgmv_gallery
-tm_import
-nc_videostatistics
-displaycontroller_debug
-st_gfi_taleo
-dialogcentral
-dscentral
-jb_metaexec_doc
-maag_cenoshop
-coreupdate
-uploadtest
-coo_facebook
-form4_teaser
-filedeletion
-form4_pages_counter
-mm_forum_blog
-browser_manual_ootb_en
-moox_feusers
-lvrandfiles
-dyncss_phpsass
-moox_slider
-ajax_calendar
-t3essentials
-moox_flexisel
-fluidcontent_fed
-layersliderlight
-onm_redirect_linkhandling
-browser_tut_map_en
-jh_ter_announcer
-wt_spamshield_formhandler
-form4_tags
-barscheduler
-fluidfoundationtheme
-form4_filecache
-form4_realurl
-dyncss_turbine
-attachmentdelete
-form4_pages
-view
-form4_faq
-t3additions
-external_link_parameter

lib/check_installation.py

@@ -1,141 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-#-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
-#-------------------------------------------------------------------------------
-
-import re
-import sys
-from colorama import Fore
-from lib.request import Request
-from lib.output import Output
-
-class Typo3_Installation:
-	"""
-	This class checks, if Typo3 is used on the domain with different approaches.
-	If Typo3 is used, a link to the default login page is shown.
-	"""
-	@staticmethod
-	def run(domain):
-		check_on_root = Typo3_Installation.check_root(domain)
-		if not check_on_root:
-			default_files = Typo3_Installation.check_default_files(domain)
-			if not default_files:
-				typo = Typo3_Installation.check_404(domain)
-
-	"""
-	This method requests the root page
-		and searches for a specific string in the response.
-		Usually there are some TYPO3 notes in the HTML comments.
-
-		If found, it searches for a Typo3 path reference
-		in order to determine the Typo3 installation path.
-	"""
-	@staticmethod
-	def check_root(domain):
-		response = Request.get_request(domain.get_name(), '/')
-		if re.search('[Tt][Yy][Pp][Oo]3', response[0]):
-			domain.set_typo3()
-			headers = Request.interesting_headers(response[1], response[2])
-			for key in headers:
-				domain.set_interesting_headers(key, headers[key])
-
-			try:
-				path = re.search('(href|src|content)=(.{0,35})(typo3temp/|typo3conf/)', response[0])
-				if not (path.groups()[1] == '"' or '"../' in path.groups()[1]):
-					real_path = (path.groups()[1].split('"')[1])
-					if 'http' in real_path:
-						domain.set_name(real_path[0:len(real_path)-1])
-					else:
-						domain.set_name(domain.get_name() + real_path[0:len(real_path)-1])
-					domain.set_path(real_path[0:len(real_path)-1])
-			except:
-				pass
-			return True
-		else:
-			return False
-
-	"""
-	This method requests different files, which are generated on installation.
-		Usually they are not deleted by admins 
-		and can be used as an indicator of a TYPO3 installation.
-	"""
-	@staticmethod
-	def check_default_files(domain):
-		files = {'/typo3_src/README.md':'[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss]',
-				'/typo3_src/README.txt':'[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss]',
-				'/typo3_src/INSTALL.txt':'INSTALLING [Tt][Yy][Pp][Oo]3',
-				'/typo3_src/INSTALL.md':'INSTALLING [Tt][Yy][Pp][Oo]3',
-				'/typo3_src/LICENSE.txt':'[Tt][Yy][Pp][Oo]3'
-			}
-
-		for path, regex in files.items():
-			try:
-				response = Request.get_request(domain.get_name(), path)
-				regex = re.compile(regex)
-				searchInstallation = regex.search(response[0])
-				installation = searchInstallation.groups()
-				domain.set_typo3()
-				return True
-			except:
-				pass
-		return False
-
-	"""
-	This method requests a site which is not available.
-		TYPO3 installations usually generate a default error page,
-		which can be used as an indicator.
-	"""
-	@staticmethod
-	def check_404(domain):
-		domain_name = domain.get_name()
-		response = Request.get_request((domain_name.split('/')[0] + '//' + domain_name.split('/')[2]), '/idontexist')
-		try:
-			regex = re.compile('[Tt][Yy][Pp][Oo]3 CMS')
-			searchInstallation = regex.search(response[0])
-			installation = searchInstallation.groups()
-			domain.set_typo3()
-			return True
-		except:
-			return False
-
-	"""
-	This method requests the default login page
-		and searches for a specific string in the title or the response.
-		If the access is forbidden (403), extension search is still possible.
-	"""
-	@staticmethod
-	def search_login(domain):
-		try:
-			response = Request.get_request(domain.get_name(), '/typo3/index.php')
-			regex = re.compile('<title>(.*)</title>', re.IGNORECASE)
-			searchTitle = regex.search(response[0])
-			title = searchTitle.groups()[0]
-
-			login_text = Fore.GREEN + domain.get_name() + '/typo3/index.php' + Fore.RESET
-			login_text += '\n | Accessible?'.ljust(30)  
-			
-			if ('TYPO3 Backend access denied: The IP address of your client' in response[0]) or (response[3] == 403):
-				login_text += (Fore.YELLOW + ' Forbidden (IP Address Restriction)' + Fore.RESET)
-			elif (('TYPO3 Login' in title) or ('TYPO3 CMS Login') in title):
-				login_text += Fore.GREEN + ' Yes' + Fore.RESET
-			else:
-				login_text = Fore.RED + 'Could not be found' + Fore.RESET
-			domain.set_login_found(login_text)
-			return True
-		except:
-			return False

lib/config.json

@@ -1 +1 @@
-{"threads": 5, "pass": "ne", "user": "No", "timeout": 10, "agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"}
+{"threads": 5, "timeout": 10, "cookie": "", "auth": "", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A"}

lib/domain.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
+# Typo3Scan - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
+# Copyright (c) 2014-2020 Jan Rude
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,84 +15,221 @@
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
 #-------------------------------------------------------------------------------
+import re
+import string
+import random
+import sqlite3
+from colorama import Fore
+import lib.request as request
+from pkg_resources import parse_version
 
-class Domain(object):
+class Domain:
-	"""
+    """
-	This class stores following information about a domain:
+    This class stores following information about a domain:
-		name: 					URL of the domain
+        name:                   URL of the domain
-		typo3:					If Typo3 is installed
+        typo3:                  If Typo3 is installed
-		typo3_version:			Typo3 Version
+        typo3_version:          Typo3 Version
-		login_found:			Determines of the default login page was found or not
+        path:                   Full path to Typo3 installation
-		extensions:				List of extensions to check for
+        installed_extensions:   List of all installed extensions
-		installed_extensions:	List of all installed extensions
+        interesting_header:     List of interesting headers
-		interesting_header:		List of interesting headers
+    """
-	"""
+    def __init__(self, name):
-	def __init__(self, name, ext_state, top=False):
+        if not ('http' in name):
-		if not ('http' in name):
+            self.__name = 'https://' + name
-			self.__name = 'http://' + name
+        else:
-		else:
+            self.__name = name
-			self.__name = name
+        self.__typo3 = False
-		self.__typo3 = False
+        self.__typo3_version = ''
-		self.__typo3_version = ''
+        self.__path = ''
-		self.__login_found = ''
+        self.__installed_extensions = {}
-		self.__path = ''
+        self.__interesting_header = {}
-		self.__extension_config = [ext_state, top]
-		self.__extensions = None
-		self.__installed_extensions = {}
-		self.__interesing_header = {}
         
-	def get_name(self):
+    def get_name(self):
-		return self.__name
+        return self.__name
 
-	def set_name(self, name):
+    def set_name(self, name):
-		self.__name = name
+        self.__name = name
 
-	def get_extensions(self):
+    def is_typo3(self):
-		return self.__extensions
+        return self.__typo3
 
-	def set_extensions(self, extensions):
+    def set_typo3(self):
-		self.__extensions = extensions
+        self.__typo3 = True
 
-	def get_extension_config(self):
+    def set_typo3_version(self, version):
-		return self.__extension_config
+        self.__typo3_version = version
 
-	def get_installed_extensions(self):
+    def get_typo3_version(self):
-		return self.__installed_extensions
+        return self.__typo3_version
 
-	def set_installed_extensions(self, extension):
+    def set_path(self, path):
-		self.__installed_extensions[extension] = False
+        self.__path = path
 
-	def set_installed_extensions_version(self, extension, ChangeLog):
+    def get_path(self):
-		self.__installed_extensions[extension] = ChangeLog
+        return self.__path
 
-	def get_typo3(self):
+    def set_interesting_headers(self, headers, cookies):
-		return self.__typo3
+        """
+        This method searches for interesing headers in the HTTP response
+            Server:           Displays the name of the server
+            X-Powered-By:     Information about Frameworks (e.g. ASP, PHP, JBoss) used by the web application
+            X-*:              Version information in other technologies
+            Via:              Informs the client of proxies through which the response was sent
+            X-Forwarded-For:  Originating IP address of a client connecting through an HTTP proxy or load balancer
+            fe_typo_user:     Frontend cookie for TYPO3
+        """
+        for header in headers:
+            if header == 'Server':
+                self.__interesting_header['Server'] = headers.get('Server')
+            elif header == 'X-Powered-By':
+                self.__interesting_header['X-Powered-By'] = headers.get('X-Powered-By')
+            elif header == 'X-Runtime':
+                self.__interesting_header['X-Runtime'] = headers.get('X-Runtime')
+            elif header == 'X-Version':
+                self.__interesting_header['X-Version'] = headers.get('X-Version')
+            elif header == 'X-AspNet-Version':
+                self.__interesting_header['X-AspNet-Version'] = headers.get('X-AspNet-Version')
+            elif header == 'Via':
+                self.__interesting_header['Via'] = headers.get('Via')
+            elif header == 'X-Forwarded-For':
+                self.__interesting_header['X-Forwarded-For'] = headers.get('X-Forwarded-For')
             
-	def set_typo3(self):
+        if 'fe_typo_user' in cookies.keys():
-		self.__typo3 = True
+            self.__interesting_header['fe_typo_user'] = cookies['fe_typo_user']
+            self.set_typo3()
 
-	def set_typo3_version(self, version):
+    def get_interesting_headers(self):
-		self.__typo3_version = version
+        return self.__interesting_header
 
-	def get_typo3_version(self):
+    def check_root(self):
-		return self.__typo3_version
+        """
+        This method requests the root page and searches for a specific string.
+            Usually there are some TYPO3 notes in the HTML comments.
+            If found, it searches for a Typo3 path reference
+            in order to determine the Typo3 installation path.
+        """
+        response = request.get_request('{}'.format(self.get_name()))
+        full_path = self.get_name()
+        self.set_interesting_headers(response['headers'], response['cookies'])
+        if re.search('powered by TYPO3', response['html']):
+            self.set_typo3()
+            path = re.search('="/?(\S*?)/?(?:typo3temp|typo3conf)/'.format(self.get_name()), response['html'])
+            if path and path.groups()[0] != '':
+                path = path.groups()[0].replace(self.get_name(), '')
+                if path != '':
+                    full_path = '{}/{}'.format(self.get_name(), path)
+        if full_path.endswith('/'):
+            full_path = full_path[:-1]
+        self.set_path(full_path)
 
-	def set_path(self, path):
+    def check_default_files(self):
-		self.__path = path
+        """
+        This method requests different files, which are generated on installation.
+            Note: They are not accessible anymore on newer Typo3 installations
+        """
+        files = {'typo3_src/README.md':'TYPO3 CMS',
+                'typo3_src/README.txt':'TYPO3 CMS',
+                'typo3_src/INSTALL.md':'INSTALLING TYPO3',
+                'typo3_src/INSTALL.txt':'INSTALLING TYPO3',
+                'typo3_src/LICENSE.txt':'TYPO3',
+                'typo3_src/CONTRIBUTING.md':'TYPO3 CMS'
+            }
+        for path, regex in files.items():
+            try:
+                response = request.get_request('{}/{}'.format(self.get_path(), path))
+                regex = re.compile(regex)
+                searchInstallation = regex.search(response['html'])
+                installation = searchInstallation.groups()
+                self.set_typo3()
+                return True
+            except:
+                pass
+        return False
 
-	def get_path(self):
+    def check_404(self):
-		return self.__path
+        """
+        This method requests a site which is not available by using a random generated string.
+            TYPO3 installations usually generate a default error page,
+            which can be used as an indicator.
+        """
+        random_string = ''.join(random.choice(string.ascii_lowercase) for i in range(10))
+        response = request.get_request('{}/{}'.format(self.get_path(), random_string))
+        search404 = re.search('[Tt][Yy][Pp][Oo]3 CMS', response['html'])
+        if search404:
+            self.set_typo3()
 
-	def get_login_found(self):
+    def search_login(self):
-		return self.__login_found
+        """
+        This method requests the default login page
+            and searches for a specific string in the title or the response.
+            If the access is forbidden (403), extension search is still possible.
+        """
+        print(' \\\n [+] Backend Login')
+        # maybe /typo3_src/typo3/index.php too?
+        response = request.get_request('{}/typo3/index.php'.format(self.get_path()))
+        searchTitle = re.search('<title>(.*)</title>', response['html'])
+        if searchTitle and 'Login' in searchTitle.group(0):
+            print('  \u2514', Fore.GREEN + '{}/typo3/index.php'.format(self.get_path()) + Fore.RESET)
+        elif ('Backend access denied: The IP address of your client' in response['html']) or (response['status_code'] == 403):
+            print('  \u251c', Fore.GREEN + '{}/typo3/index.php'.format(self.get_path()) + Fore.RESET)
+            print('  \u2514', Fore.YELLOW + 'But access is forbidden (IP Address Restriction)' + Fore.RESET)
+        else:
+            print('  \u251c', Fore.RED + 'Could not be found' + Fore.RESET)
 
-	def set_login_found(self, path):
+    def search_typo3_version(self):
-		self.__login_found = path
+        """
+        This methos will search for version information.
+            The exact version can be found in the ChangeLog, therefore it will be requested first.
+            Less specific version information can be found in the NEWS or INSTALL file. 
+        """           
+        files = {'/typo3_src/ChangeLog': '[Tt][Yy][Pp][Oo]3 (\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',  
+                '/ChangeLog': '[Tt][Yy][Pp][Oo]3 (\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',
+                '/typo3_src/typo3/sysext/install/Start/Install.php': '(?:CMS |typo3_src-)(\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',
+                '/typo3/sysext/install/Start/Install.php': '(?:CMS |typo3_src-)(\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',
+                '/typo3_src/typo3/sysext/backend/composer.json': '"typo3/cms-core": "(\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)"',
+                '/typo3_src/NEWS.txt': 'http://wiki.typo3.org/TYPO3_(\d{1,2}\.\d{1,2})', 
+                '/typo3_src/NEWS.md': '[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}\.\d{1,2}) - WHAT\'S NEW', 
+                '/NEWS.txt': 'http://wiki.typo3.org/TYPO3_(\d{1,2}\.\d{1,2})', 
+                '/NEWS.md': '[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}\.\d{1,2}) - WHAT\'S NEW',
+                '/INSTALL.md': '[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}(.\d{1,2})?)',
+                '/INSTALL.txt': '[Tt][Yy][Pp][Oo]3 v(\d{1})'
+                }
 
-	def set_interesting_headers(self, header_key, header_value):
+        version = None
-		self.__interesing_header[header_key] = header_value
+        for path, regex in files.items():
+            response = request.version_information(self.get_path()+path, regex)
+            if not (response is None) and (version is None or (len(response) > len(version))):
+                version = response
+                version_path = path
 
-	def get_interesting_headers(self):
+        print('\n [+] Version Information')
-		return self.__interesing_header
+        if not (version is None):
+            print('  \u251c {}'.format(Fore.GREEN + version + Fore.RESET))
+            print('  \u251c see: {}{}'.format(self.get_path(), version_path))
+            if len(version) == 3:
+                print('  \u251c Could not identify exact version.')
+                react = input('  \u251c Do you want to print all vulnerabilities for branch {}? (y/n): '.format(version))
+                if react.startswith('y'):
+                    version = version + '.0'
+                else:
+                	return False
+            print('  \u2514 Known vulnerabilities\n     \\')
+            # sqlite stuff
+            conn = sqlite3.connect('lib/typo3scan.db')
+            c = conn.cursor()
+            c.execute('SELECT advisory, vulnerability, subcomponent, affected_version_max, affected_version_min FROM core_vulns WHERE (?<=affected_version_max AND ?>=affected_version_min)', (version, version,))
+            data = c.fetchall()
+            if not data:
+                print('     \u251c None.')
+            else:
+                for vuln in data:
+                    # maybe instead use this: https://oraerr.com/database/sql/how-to-compare-version-string-x-y-z-in-mysql-2/
+                    if parse_version(version) <= parse_version(vuln[3]):
+                        print('     [!] {}'.format(Fore.RED + vuln[0] + Fore.RESET))
+                        print('      \u251c Vulnerability Type:'.ljust(29), vuln[1])
+                        print('      \u251c Subcomponent:'.ljust(29), vuln[2])
+                        print('      \u2514 Affected Versions:'.ljust(29), '{} - {}\n'.format(vuln[3], vuln[4]))
+        else:
+            print('  \u251c', Fore.RED + 'No version information found' + Fore.RESET)

lib/extensions.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
 # Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
+# Copyright (c) 2014-2020 Jan Rude
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,86 +15,102 @@
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
 #-------------------------------------------------------------------------------
 
-import os.path
+import sqlite3
-import json
+from colorama import Fore
-from lib.request import Request
+import lib.request as request
-from lib.output import Output
 from lib.thread_pool import ThreadPool
+from pkg_resources import parse_version
 
 class Extensions:
-	"""
+    """
-	Extension class
+    Extension class
-	"""
+    """
-	def __init__(self, ext_state, top, path):
+    def __init__(self):
-		self.__ext_state = ext_state
+        pass
-		self.__top = top
-		self.__path = path
 
-	def load_extensions(self):
+    def search_extension(self, domain, extensions, threads):
-		"""
+        """
-			This method loads the defined extensions from the extension file.
+            This method loads the extensions from the database and searches for installed extensions.
-			IF the extension file is not found, an error is raised.
+                /typo3conf/ext/:        Local installation path. This is where extensions usually get installed.
-		"""
+                /typo3/ext/:            Global installation path (not used atm)
-		extensions = []
+                /typo3/sysext/:            Extensions shipped with core
-		for state in self.__ext_state:
+        """
-			ext_file = state + '_extensions'
+        found_extensions = {}
-			if not os.path.isfile(os.path.join(self.__path, 'extensions', ext_file)):
+        thread_pool = ThreadPool()
-				raise Exception("\n\nCould not find extension file " + ext_file + '!\nTry --update')
+        for ext in extensions:
+            thread_pool.add_job((request.head_request, ('{}/typo3conf/ext/{}/'.format(domain, ext))))
+            thread_pool.add_job((request.head_request, ('{}/typo3/sysext/{}/'.format(domain, ext))))
+            #thread_pool.add_job((request.head_request, ('{}/typo3/ext/{}/'.format(domain, ext))))
+        thread_pool.start(threads)
 
-			with open(os.path.join(self.__path, 'extensions', ext_file), 'r') as f:
+        for installed_extension in thread_pool.get_result():
-				count = 0
+            name = installed_extension[1][:-1]
-				for extension in f:
+            name = name[name.rfind('/')+1:]
-					if not(self.__top is None):
+            found_extensions[name] = {'url':installed_extension[1], 'version': None, 'file': None}
-						if count < self.__top:
+        return found_extensions
-							extensions.append(extension.split('\n')[0])
-							count += 1
-					else:
-						extensions.append(extension.split('\n')[0])
-				f.close()
-		return extensions
 
-	def search_extension(self, domain, extensions):
+    def search_ext_version(self, found_extensions, threads):
-		"""
+        """
-			This method searches for installed extensions.
+            This method adds a job for every installed extension.
-				/typo3conf/ext/:		Local installation path. This is where extensions get usually installed.
+            The goal is to find a file with version information.
-				/typo3/ext/:			Global installation path (not used atm)
+        """
-				/typo3/sysext/:			Extensions shipped with core (not used atm)
+        thread_pool = ThreadPool()
-		"""
+        for extension,values in found_extensions.items():
-		config = json.load(open(os.path.join(self.__path, 'lib', 'config.json')))
+            thread_pool.add_job((request.version_information, (values['url'] + 'Documentation/ChangeLog/Index.rst', None)))
-		thread_pool = ThreadPool()
+            thread_pool.add_job((request.version_information, (values['url'] + 'Documentation/Settings.cfg', None)))
-		for ext in extensions:
+            thread_pool.add_job((request.version_information, (values['url'] + 'Documentation/Settings.yml', None)))
-			thread_pool.add_job((Request.head_request, (domain.get_name(), '/typo3conf/ext/' + ext)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'Settings.yml', None)))
-			#thread_pool.add_job((Request.head_request, (domain.get_name(), '/typo3/ext/' + ext)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'Documentation/Index.rst', None)))
-			#thread_pool.add_job((Request.head_request, (domain.get_name(), '/typo3/sysext/' + ext)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'composer.json', '(?:"dev-master":|"version":)\s?"([0-9]+\.[0-9]+\.[0-9x][0-9x]?)')))
-		thread_pool.start(config['threads'])
+            thread_pool.add_job((request.version_information, (values['url'] + 'Index.rst', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'ChangeLog', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'CHANGELOG.md', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'ChangeLog.txt', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'Readme.txt', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'README.md', None)))
+            thread_pool.add_job((request.version_information, (values['url'] + 'README.rst', None)))
         
-		for installed_extension in thread_pool.get_result():
+        thread_pool.start(threads, version_search=True)
-			domain.set_installed_extensions(installed_extension[1][1])
 
-	def search_ext_version(self, domain, extension_dict):
+        for version_path in thread_pool.get_result():
-		"""
+            path = version_path[0][0]
-			This method adds a job for every installed extension.
+            version = version_path[1]
-			The goal is to find a ChangeLog or Readme in order to determine the version.
+            name = version_path[0][0]
-		"""
+            if 'Documentation/' in name:
-		config = json.load(open('lib/config.json'))
+                name = name[:name.rfind('Documentation/')+1]
-		thread_pool = ThreadPool()
+            name = name[name.find('ext/')+4:name.rfind('/')]
-		for extension_path in extension_dict:
+            found_extensions[name]['version'] = version
-			thread_pool.add_job((Request.head_request, (domain.get_name(), extension_path + '/ChangeLog')))
+            found_extensions[name]['file'] = path
-			thread_pool.add_job((Request.head_request, (domain.get_name(), extension_path + '/ChangeLog.txt')))
+        return found_extensions
-			thread_pool.add_job((Request.head_request, (domain.get_name(), extension_path + '/Readme.txt')))
-			thread_pool.add_job((Request.head_request, (domain.get_name(), extension_path + '/README.md')))
-			thread_pool.add_job((Request.head_request, (domain.get_name(), extension_path + '/README.rst')))
 
-		thread_pool.start(config['threads'], True)
 
-		for changelog_path in thread_pool.get_result():
+    def output(self, extension_dict, database):
-			ext, path = self.parse_extension(changelog_path)
+        conn = sqlite3.connect(database)
-			domain.set_installed_extensions_version(path, ext[4])
+        c = conn.cursor()
-
+        print('\n\n [+] Extension information\n  \\')
-	def parse_extension(self, path):
+        for extension,info in extension_dict.items():
-		ext = (path[1][1]).split('/')
+            c.execute('SELECT title FROM extensions where extensionkey=?', (extension,))
-		path = '/' + ext[1] + '/' + ext[2] + '/' + ext[3]
+            title = c.fetchone()[0]
-		return (ext, path)
+            print('  [+] Name: {}'.format(Fore.GREEN + extension + Fore.RESET))
+            print('   \u251c Title: {}'.format(title))
+            if info['version']:
+                c.execute('SELECT advisory, vulnerability, affected_version_max, affected_version_min FROM extension_vulns WHERE (extensionkey=? AND ?<=affected_version_max AND ?>=affected_version_min)', (extension, info['version'], info['version'],))
+                data = c.fetchall()
+                print('   \u251c Version: {}'.format(Fore.GREEN + info['version'] + Fore.RESET))
+                if data:
+                    print('   \u251c see: {}'.format(info['file']))
+                    print('   \u2514 Known vulnerabilities\n      \\')
+                    for vuln in data:
+                        if parse_version(info['version']) <= parse_version(vuln[2]):
+                            print('     [!] {}'.format(Fore.RED + vuln[0] + Fore.RESET))
+                            print('      \u251c Vulnerability Type:'.ljust(29), vuln[1])
+                            print('      \u2514 Affected Versions:'.ljust(29), '{} - {}'.format(vuln[2], vuln[3]))
+                else:
+                    print('   \u2514 see: {}'.format(info['file']))
+            else:
+            	print('   \u2514 Version: -unknown-')
+            print()
+        conn.close()

lib/initdb.py

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#-------------------------------------------------------------------------------
+# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
+# Copyright (c) 2014-2020 Jan Rude
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+#-------------------------------------------------------------------------------
+import sqlite3, os.path
+
+class DB_Init:
+    """
+    This class will empty the database, create tables and insert User-Agents
+    """
+    def __init__(self):
+        database = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'typo3scan.db')
+        try:
+            conn = sqlite3.connect(database)
+            c = conn.cursor()
+            
+            # Delete all tables
+            c.execute('''DROP TABLE IF EXISTS extensions''')
+            c.execute('''DROP TABLE IF EXISTS extension_vulns''')
+            c.execute('''DROP TABLE IF EXISTS core_vulns''')
+            c.execute('''DROP TABLE IF EXISTS settings''')
+            conn.commit()
+
+            # Create table extensions
+            c.execute('''CREATE TABLE IF NOT EXISTS extensions
+                         (title text, extensionkey text PRIMARY KEY, description text, version text, state text)''')
+
+            # Create table extension_vulns
+            c.execute('''CREATE TABLE IF NOT EXISTS extension_vulns
+                         (advisory text, extensionkey text, vulnerability text, branch_max integer, affected_version_max text, branch_max integer, affected_version_min text)''')
+
+            # Create table core_vulns
+            c.execute('''CREATE TABLE IF NOT EXISTS core_vulns
+                         (advisory text, vulnerability text, subcomponent text, branch_max integer, affected_version_max text, branch_max integer, affected_version_min text, cve text)''')
+
+            # Create table UserAgents
+            c.execute('''CREATE TABLE IF NOT EXISTS UserAgents
+                         (userAgent text)''')
+            conn.commit()
+
+            # add some User-Agents from http://www.useragentstring.com/pages/useragentstring.php
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux i686; rv:64.0) Gecko/20100101 Firefox/64.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux i586; rv:63.0) Gecko/20100101 Firefox/63.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.2; WOW64; rv:63.0) Gecko/20100101 Firefox/63.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/44.0.2403.155 Safari/537.36',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14931',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Chrome (AppleWebKit/537.1; Chrome50.0; Windows NT 6.3) AppleWebKit/537.36 (KHTML like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.9200',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20121202 Firefox/17.0 Iceweasel/17.0.1',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1 Iceweasel/15.0.1',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux i686; rv:15.0) Gecko/20100101 Firefox/15.0.1 Iceweasel/15.0.1',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120724 Debian Iceweasel/15.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0 Iceweasel/15.0',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Opera/9.80 (Macintosh; Intel Mac OS X 10.14.1) Presto/2.12.388 Version/12.16',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Windows NT 6.0; rv:2.0) Gecko/20100101 Firefox/4.0 Opera 12.14',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0) Opera 12.14',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25',))
+            c.execute('INSERT INTO UserAgents VALUES (?)', ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2',))
+            conn.commit()
+
+        except sqlite3.Error as e:
+            if conn:
+                conn.rollback()
+            print(e)
+            sys.exit(-1)
+
+        finally:
+            if conn:
+                conn.close()
+            print('\n[+] Database resetted')
+            print('[!] Please update (-u) the database before using Typo3Scan.\n')

lib/output.py

@@ -1,63 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-#-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
-#-------------------------------------------------------------------------------
-
-from colorama import Fore
-
-class Output:
-	"""
-		This class handles the output
-	"""
-	def __init__(self):
-		pass
-
-	def typo3_installation(domain):
-		"""
-			If TYPO3 is installed and the backend login was found, a link to a backend is printed.
-			Additionally, if the version search was successful, the version and a link to cvedetails is given.
-		"""
-		print('')
-		print('[+] Typo3 backend login:'.ljust(30) + domain.get_login_found())
-		if (domain.get_typo3_version() != 'could not be determined'):
-			print('[+] Typo3 version:'.ljust(30) + Fore.GREEN + domain.get_typo3_version() + Fore.RESET)
-			print(' | known vulnerabilities:'.ljust(30) + Fore.GREEN + 'http://www.cvedetails.com/version-search.php?vendor=&product=Typo3&version=' + domain.get_typo3_version() + Fore.RESET)
-		else:
-			print('[+] Typo3 version:'.ljust(30) + Fore.RED + domain.get_typo3_version() + Fore.RESET)
-		print('')
-
-	def interesting_headers(name, value):
-		"""
-			This method prints interesting headers
-		"""
-		string = '[!] ' + name + ':'
-		print(string.ljust(30) + value)
-
-	def extension_output(path, extens):
-		"""
-			This method prints every found extension.
-			If a Readme or ChangeLog is found, it will print a link to the file.
-		"""
-		if not extens:
-			print(Fore.RED + ' | No extension found' + Fore.RESET)
-		else:
-			for extension in extens:
-				print(Fore.BLUE + '\n[+] Name: ' + extension.split('/')[3] + '\n' + "-"* 31  + Fore.RESET)
-				print(' | Location:'.ljust(16) + path + extension)
-				if not (extens[extension] == False):
-					print(' | ' + extens[extension].split('.')[0] + ':'.ljust(4) + (path + extension + '/'+ extens[extension]))

lib/request.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
 # Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
+# Copyright (c) 2014-2020 Jan Rude
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,127 +15,120 @@
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
 #-------------------------------------------------------------------------------
 
 import re
+import os.path
 import json
 import requests
+from colorama import Fore
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
 requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
-from colorama import Fore
-from lib.output import Output
 
-class Request:
+def get_request(url):
-	"""
+    """
-		This class is used to make all server requests
+    All GET requests are done in this method.
-	"""
+        This method is not used, when searching for extensions and their Readmes/ChangeLogs
-	@staticmethod
+        There are three error types which can occur:
-	def get_request(domain_name, path):
+            Connection timeout
-		"""
+            Connection error
-		All GET requests are done in this method.
+            anything else
-			This method is not used, when searching for extensions and their Readmes/ChangeLogs
+    """
-			There are three error types which can occur:
+    config = json.load(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.json')))
-				Connection timeout
+    timeout = config['timeout']
-				Connection error
+    auth = config['auth']
-				anything else
+    cookie = config['cookie']
-		"""
+    custom_headers = {'User-Agent' : config['User-Agent']}
-		try:
+    try:
-			config = json.load(open('lib/config.json'))
+        if cookie != '':
-			cookie = {config['cookie'].split('=')[0]:config['cookie'].split('=')[1]}
+            name = cookie.split('=')[0]
-			r = requests.get(domain_name + path, timeout=config['timeout'], headers={'User-Agent' : config['agent']}, cookies=cookie, auth=(config['user'], config['pass']), verify=False)
+            value = cookie.split('=')[1]
-			httpResponse = str((r.text).encode('utf-8'))
+            custom_headers[name] = value
-			headers = r.headers
+        response = {}
-			cookies = r.cookies
+        if auth != '':
-			status_code = r.status_code
+            r = requests.get(url, timeout=config['timeout'], headers=custom_headers, auth=(auth.split(':')[0], auth.split(':')[1]), verify=False)
-			response = [httpResponse, headers, cookies, status_code]
+        else:
-			return response
+            r = requests.get(url, timeout=config['timeout'], headers=custom_headers, verify=False)
-		except requests.exceptions.Timeout:
+        response['status_code'] = r.status_code
-			print(e)
+        response['html'] = r.text
-			print(Fore.RED + '[x] Connection timed out' + Fore.RESET)
+        response['headers'] = r.headers
-		except requests.exceptions.ConnectionError as e: 
+        response['cookies'] = r.cookies
-			print(e)
+        return response
-			print(Fore.RED + '[x] Connection error\n | Please make sure you provided the right URL' + Fore.RESET)
+    except requests.exceptions.Timeout:
-		except requests.exceptions.RequestException as e:
+        print(e)
-			print(Fore.RED + str(e) + Fore.RESET)
+        print(Fore.RED + '[x] Connection timed out' + Fore.RESET)
+    except requests.exceptions.ConnectionError as e: 
+        print(e)
+        print(Fore.RED + '[x] Connection error\n | Please make sure you provided the right URL' + Fore.RESET)
+        exit(-1)
+    except requests.exceptions.RequestException as e:
+        print(Fore.RED + str(e) + Fore.RESET)
 
-	@staticmethod
+def head_request(url):
-	def head_request(domain_name, path):
+    """
-		"""
+    All HEAD requests are done in this method.
-		All HEAD requests are done in this method.
+        HEAD requests are used when searching for extensions and their Readmes/ChangeLogs
-			HEAD requests are used when searching for extensions and their Readmes/ChangeLogs
+        There are three error types which can occur:
-			There are three error types which can occur:
+            Connection timeout
-				Connection timeout
+            Connection error
-				Connection error
+            anything else
-				anything else
+    """
-		"""
-		try:
-			config = json.load(open('lib/config.json'))
-			r = requests.head(domain_name + path, timeout=config['timeout'], headers={'User-Agent' : config['agent']}, auth=(config['user'], config['pass']), allow_redirects=False, verify=False)
-			status_code = str(r.status_code)
-			if status_code == '405':
-				print("WARNING, (HEAD) method not allowed!!")
-				exit(-1)
-			return status_code
-		except requests.exceptions.Timeout:
-			print(Fore.RED + '[x] Connection timed out' + Fore.RESET)
-		except requests.exceptions.ConnectionError as e: 
-			print(Fore.RED + '[x] Connection aborted.\n    Please make sure you provided the right URL' + Fore.RESET)
-		except requests.exceptions.RequestException as e:
-			print(Fore.RED + str(e) + Fore.RESET)
     
-	@staticmethod
+    config = json.load(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.json')))
-	def interesting_headers(headers, cookies):
+    timeout = config['timeout']
-		"""
+    auth = config['auth']
-		This method searches for interesing headers in the HTTP response.
+    cookie = config['cookie']
-			Server:			Displays the name of the server
+    custom_headers = {'User-Agent' : config['User-Agent']}
-			X-Powered-By:	Information about Frameworks (e.g. ASP, PHP, JBoss) used by the web application
+    try:
-			X-*:			Version information in other technologies
+        if cookie != '':
-			Via:			Informs the client of proxies through which the response was sent.
+            name = cookie.split('=')[0]
-			be_typo_user:	Backend cookie for TYPO3
+            value = cookie.split('=')[1]
-			fe_typo_user:	Frontend cookie for TYPO3
+            custom_headers[name] = value
-		"""
+        if auth != '':
-		found_headers = {}
+            r = requests.head(url, timeout=config['timeout'], headers=custom_headers, auth=(auth.split(':')[0], auth.split(':')[1]), verify=False)
-		for header in headers:
+        else:
-			if header == 'server':
+            r = requests.head(url, timeout=config['timeout'], headers=custom_headers, allow_redirects=False, verify=False)
-				found_headers['Server'] = headers.get('server')
+        status_code = str(r.status_code)
-			elif header == 'x-powered-by':
+        if status_code == '405':
-				found_headers['X-Powered-By'] = headers.get('x-powered-by')
+            print('[x] WARNING: \'HEAD\' method not allowed!')
-			elif header == 'x-runtime':
+            exit(-1)
-				found_headers['X-Runtime'] = headers.get('x-runtime')
+        return status_code
-			elif header == 'x-version':
+    except requests.exceptions.Timeout:
-				found_headers['X-Version'] = headers.get('x-version')
+        print(Fore.RED + '[x] Connection timed out' + Fore.RESET)
-			elif header == 'x-aspnet-version':
+    except requests.exceptions.ConnectionError as e: 
-				found_headers['X-AspNet-Version'] = headers.get('x-aspnet-version')
+        print(Fore.RED + '[x] Connection aborted.\n    Please make sure you provided the right URL' + Fore.RESET)
-			elif header == 'via':
+    except requests.exceptions.RequestException as e:
-				found_headers['Via'] = headers.get('via')
+        print(Fore.RED + str(e) + Fore.RESET)
-		try:
-			typo_cookie = cookies['be_typo_user']
-			found_headers['be_typo_user'] = typo_cookie
-		except:
-			pass
-		try:
-			typo_cookie = cookies['fe_typo_user']
-			found_headers['fe_typo_user'] = typo_cookie
-		except:
-			pass
-		return found_headers
 
-	@staticmethod
+def version_information(url, regex):
-	def version_information(domain_name, path, regex):
+    """
-		"""
+        This method is used for version search only.
-			This method is used for version search only.
+        It performs a GET request, if the response is 200 - Found, it reads the first 400 bytes the response only,
-			It performs a GET request, if the response is 200 - Found, it reads the first 400 bytes the response only,
+        because usually the TYPO3 version is in the first few lines of the response.
-			because usually the TYPO3 version is in the first few lines of the response.
+    """
-		"""
+    if regex is None:
-		config = json.load(open('lib/config.json'))
+        regex = '([0-9]+\.[0-9]+\.[0-9x][0-9x]?)'
-		r = requests.get(domain_name + path, stream=True, timeout=config['timeout'], headers={'User-Agent' : config['agent']}, auth=(config['user'], config['pass']), verify=False)
+    config = json.load(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.json')))
-		if r.status_code == 200:
+    timeout = config['timeout']
-			try:
+    auth = config['auth']
-				for content in r.iter_content(chunk_size=400, decode_unicode=False):
+    cookie = config['cookie']
-					regex = re.compile(regex)
+    custom_headers = {'User-Agent' : config['User-Agent']}
-					search = regex.search(str(content))
+    if cookie != '':
-					version = search.groups()[0]
+        name = cookie.split('=')[0]
-					return version
+        value = cookie.split('=')[1]
-			except:
+        custom_headers[name] = value
-				return None
+    if auth != '':
+        r = requests.get(url, stream=True, timeout=config['timeout'], headers=custom_headers, auth=(auth.split(':')[0], auth.split(':')[1]), verify=False)
+    else:
+        r = requests.get(url, stream=True, timeout=config['timeout'], headers=custom_headers, verify=False)
+    if r.status_code == 200:
+        try:
+            for content in r.iter_content(chunk_size=400, decode_unicode=False):
+                search = re.search(regex, str(content))
+                version = search.group(1)
+                r.close()
+                return version
+        except:
+            r.close()
+            return None

lib/thread_pool.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
 # Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
+# Copyright (c) 2014-2020 Jan Rude
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,12 +15,15 @@
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
 #-------------------------------------------------------------------------------
 
 import threading
 from queue import Queue
+from progressbar import Bar, AdaptiveETA, Percentage, ProgressBar
 
+bar = None
+number = 1
 class ThreadPoolSentinel:
 	pass
 
@@ -34,6 +37,8 @@ class ThreadPool:
 		thread_list:			List of worker threads
 	"""
 	def __init__(self):
+		global number
+		number = 1
 		self.__work_queue = Queue()
 		self.__result_queue = Queue()
 		self.__active_threads = 0
@@ -51,35 +56,33 @@ class ThreadPool:
 				active_threads -= 1
 				self.__result_queue.task_done()
 				continue
-
 			else: # Getting an actual result
 				self.__result_queue.task_done()
 				yield result
 
 	def start(self, threads, version_search=False):
+		global bar
+		toolbar_width = (self.__work_queue).qsize()
+		widgets = ['  \u251c Processed: ', Percentage(),' ', Bar(),' ', AdaptiveETA()]
+		bar = ProgressBar(widgets=widgets, maxval=toolbar_width).start()
 		if self.__active_threads:
 			raise Exception('Threads already started.')
-
+		try:
-		if not version_search:
+			# Create thread pool
-			 # Create thread pool
 			for _ in range(threads):
 				worker = threading.Thread(
 					target=_work_function,
-					args=(self.__work_queue, self.__result_queue))
+					args=(self.__work_queue, self.__result_queue, version_search))
-				worker.start()
+				worker.daemon = True
-				self.__thread_list.append(worker)
-				self.__active_threads += 1
-		else:
-			for _ in range(threads):
-				worker = threading.Thread(
-					target=_work_function_version,
-					args=(self.__work_queue, self.__result_queue))
 				worker.start()
 				self.__thread_list.append(worker)
 				self.__active_threads += 1
 
-		# Put sentinels to let the threads know when there's no more jobs
+			# Put sentinels to let the threads know when there's no more jobs
-		[self.__work_queue.put(ThreadPoolSentinel()) for worker in self.__thread_list]
+			[self.__work_queue.put(ThreadPoolSentinel()) for worker in self.__thread_list]
+		except KeyboardInterrupt:
+			print('\nReceived keyboard interrupt.\nQuitting...')
+			exit(-1)
 
 	def join(self): # Clean exit
 		self.__work_queue.join()
@@ -87,11 +90,11 @@ class ThreadPool:
 		self.__active_threads = 0
 		self.__result_queue.join()
 
-def _work_function(job_q, result_q):
+def _work_function(job_q, result_q, version_search):
 	"""Work function expected to run within threads."""
+	global number
 	while True:
 		job = job_q.get()
-
 		if isinstance(job, ThreadPoolSentinel): # All the work is done, get out
 			result_q.put(ThreadPoolSentinel())
 			job_q.task_done()
@@ -100,33 +103,17 @@ def _work_function(job_q, result_q):
 		function = job[0]
 		args = job[1]
 		try:
-			result = function(*args)
+			if version_search:
+				result = function(*args)
+			else:
+				result = function(args)
+			if not version_search and (result == '403' or result == '200'):
+				result_q.put((job))
+			elif version_search and result:
+				result_q.put((args, result))
 		except Exception as e:
 			print(e)
-		else:
-			if result == ('301' or '200' or '403'):
-				result_q.put((job))
-		finally:
-			job_q.task_done()
-
-def _work_function_version(job_q, result_q):
-	"""Work function expected to run within threads."""
-	while True:
-		job = job_q.get()
-
-		if isinstance(job, ThreadPoolSentinel): # All the work is done, get out
-			result_q.put(ThreadPoolSentinel())
-			job_q.task_done()
-			break
-
-		function = job[0]
-		args = job[1]
-		try:
-			result = function(*args)
-		except Exception as e:
-			print(e)
-		else:
-			if result == ('200'):
-				result_q.put((job))
 		finally:
+			bar.update(number)
+			number = number+1
 			job_q.task_done()

lib/tor.py

@@ -1,97 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
-#-------------------------------------------------------------------------------
-
-import socket
-import os, sys
-import re
-from colorama import Fore
-from lib.request import Request
-
-try:
-	import socks
-except:
-	print(Fore.RED + 'The module \'SocksiPy\' is not installed.')
-	if sys.platform.startswith('linux'):
-		print('Please install it with: sudo apt-get install python-socksipy' + Fore.RESET)
-	else:
-		print('You can download it from https://code.google.com/p/socksipy-branch/' + Fore.RESET)
-	sys.exit(-2)
-
-class Tor:
-	"""
-	This class initiates the usage of TOR for all requests
-		port:	TOR port
-	"""
-	def __init__(self, port=9150):
-		self.__port = port
-
-	def start_daemon(self):
-		"""
-			If the OS is linux, start TOR deamon.
-			If not, user needs to start it manually 
-		"""
-		if sys.platform.startswith('linux'):
-			os.system('service tor start')
-		elif sys.platform.startswith('win32') or sys.platform.startswith('cygwin'):
-			print('Please make sure TOR is running...')
-		else:
-			print('You are using', sys.platform, ', which is not supported (yet).')
-			sys.exit(-2)
-		
-	# Using TOR for all connections
-	def connect(self):
-		"""
-			This method checks the connection with TOR.
-				If TOR is not used, the program will exit
-		"""
-		print('\nChecking connection...')
-		socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', self.__port, True)
-		socks.socket.setdefaulttimeout(20)
-		socket.socket = socks.socksocket
-		try:
-			request = Request.get_request('https://check.torproject.org', '/')
-			response = request[0]
-		except:
-			print('Failed to connect through TOR!')
-			print('Please make sure your configuration is right!\n')
-			sys.exit(-2)
-		try:
-			regex = re.compile('Congratulations. This browser is configured to use Tor.')
-			searchVersion = regex.search(response)
-			version = searchVersion.groups()
-			print('Connection to TOR established')
-			regex = re.compile("(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})")
-			searchIP = regex.search(response)
-			IP = searchIP.groups()[0]
-			print('Your IP is: ', IP)
-		except Exception as e:
-			print(e)
-			print('It seems like TOR is not used.\nAborting...\n')
-			sys.exit(-2)
-
-	def stop(self):
-		"""
-			This method stops the TOR deamon if running under linux
-		"""
-		print('\n')
-		if sys.platform.startswith('linux'):
-			os.system('service tor stop')
-		elif sys.platform.startswith('win32') or sys.platform.startswith('cygwin'):
-			print('You can stop TOR now...')

lib/update.py

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
 # Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
+# Copyright (c) 2014-2020 Jan Rude
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,131 +15,310 @@
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
 #-------------------------------------------------------------------------------
 
-import os, sys, gzip, urllib.request, inspect
+import os.path
-from collections import OrderedDict
+from pkg_resources import parse_version
 import xml.etree.ElementTree as ElementTree
+import re, os, sys, gzip, urllib.request, sqlite3, requests
+from requests.packages.urllib3.exceptions import InsecureRequestWarning
+requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
+
+database = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'typo3scan.db')
+conn = sqlite3.connect(database)
+c = conn.cursor()
 
 class Update:
-	"""
+    """
-	This class updates the Typo3 extensions
+    This class updates the extension and vulnerability database
 
-	It will download the extension file from the official repository, 
+    It will download the extension file from the official repository, 
-	unpack it and sort the extensions in different files
+    unpack it and insert the extensions in the database.
-	"""
+    Vulnerabilities will be parsed from the official homepage.
-	def __init__(self, path):
+    """
-		print('')
+    def __init__(self):
-		self.__path = path
+        self.load_core_vulns()
-		self.download_ext()
+        self.download_ext()
-		self.generate_list()
+        self.load_extensions()
+        self.load_extension_vulns()
 
-	# Progressbar
+    def load_core_vulns(self):
-	def dlProgress(self, count, blockSize, totalSize):
+        """
-		"""
+            Grep the CORE vulnerabilities from the security advisory website
-			Progressbar for extension download
-		"""
-		percent = int(count*blockSize*100/totalSize)
-		sys.stdout.write('\r[+] Downloading extentions: ' + '%d%%' % percent)
-		sys.stdout.flush()
             
-	# Download extensions from typo3 repository
+            Search for advisories and maximum pages
-	def download_ext(self):
+            Request every advisory and get:
-		"""
+                Advisory Title
-			Download extensions from server and unpack the ZIP
+                Vulnerability Type
-		"""
+                Subcomponent(s)
-		try:
+                Affected Versions
-			# Maybe someday we need to use mirrors: https://repositories.typo3.org/mirrors.xml.gz
+                CVE Numbers
-			urllib.request.urlretrieve('https://typo3.org/fileadmin/ter/extensions.xml.gz', 'extensions.xml.gz', reporthook=self.dlProgress)
+        """
-			with gzip.open('extensions.xml.gz', 'rb') as infile:
+        print('\n[+] Searching for new CORE vulnerabilities...')
-				with open('extensions.xml', 'wb') as outfile:
+        update_counter = 0
-					for line in infile:
+        response = requests.get('https://typo3.org/help/security-advisories/typo3-cms/1')
-						outfile.write(line)
+        pages = re.findall('<a class=\"page-link\" href=\"/help/security-advisories/typo3-cms/([0-9]+)\">', response.text)
-			infile.close()
+        last_page = int(pages[-1])
-			outfile.close()
-		except Exception as e:
-			print ('\n', e)
 
-	# Parse extension file and save extensions in files
+        for current_page in range(1, last_page+1):
-	def generate_list(self):
+            print(' \u251c Page {}/{}'.format(current_page, last_page))
-		"""
+            response = requests.get('https://typo3.org/help/security-advisories/typo3-cms/{}'.format(current_page), timeout=6)
-			Parse the extension file and 
+            advisories = re.findall('TYPO3-CORE-SA-[0-9][0-9][0-9][0-9]-[0-9][0-9][0-9]', response.text)
-			sort them according to state and download count
+            for advisory in advisories:
-		"""
+                vulnerabilities = []
-		experimental = {} # 'experimental' and 'test'
+                affected_version_max = '0.0.0' 
-		alpha = {}
+                affected_version_min = '0.0.0'
-		beta = {}
+                html = requests.get('https://typo3.org/security/advisory/{}'.format(advisory.lower()))
-		stable = {}
+                beauty_html = html.text
-		outdated = {} # 'obsolete' and 'outdated'
+                beauty_html = beauty_html[beauty_html.index('Component Type'):]
-		allExt = {}
+                beauty_html = beauty_html[:beauty_html.index('General ')]
+                beauty_html = beauty_html.replace('\xa0', ' ')
+                beauty_html = beauty_html.replace('</strong>', '')
+                beauty_html = beauty_html.replace('&nbsp;', ' ')
+                beauty_html = beauty_html.replace('&amp;', '&')
 
-		print ('\n[+] Parsing file...')
+                # set as global versions
-		tree = ElementTree.parse('extensions.xml') 
+                advisory_items = {}
-		root = tree.getroot()
+                subcomponents = re.findall('([sS]ubcomponent\s?#?[0-9]?:\s?(.*?))<', beauty_html)
-		extension = 0
+                # if no subcomponent / CORE vuln
-		# for every extension in file
+                if len(subcomponents) == 0:
-		for child in root:
+                    missed = re.search('Component Type:\s?(.*?)<', beauty_html).group(1)
-			# insert every extension in "allExt" dictionary
+                    advisory_items[missed] = []
-			allExt.update({child.get('extensionkey'):child[0].text})
+                    advisory_items[missed].append(beauty_html)
-			# and search the last version entry
+                subcomponents.reverse()
-			version = 0
+                try:
-			for version_entry in root[extension].iter('version'):
+                    for subcomponent in subcomponents:
-				version +=1
+                        index = beauty_html.rfind(subcomponent[0])
-			# get the state of the latest version
+                        item_text = subcomponent[1]
-			state = (str(root[extension][version][2].text)).lower()
+                        if item_text in advisory_items:
-			if state == 'experimental' or state == 'test':
+                            item_text = item_text + ' (2)'
-				experimental.update({child.get('extensionkey'):child[0].text})
+                        advisory_items[item_text] = []
-			elif state == 'alpha':
+                        advisory_items[item_text].append(beauty_html[index:])
-				alpha.update({child.get('extensionkey'):child[0].text})
+                        beauty_html = beauty_html[:index]
-			elif state == 'beta':
-				beta.update({child.get('extensionkey'):child[0].text})
-			elif state == 'stable':
-				stable.update({child.get('extensionkey'):child[0].text})
-			elif state == 'obsolete' or state == 'outdated':
-				outdated.update({child.get('extensionkey'):child[0].text})
-			extension+=1
 
-		# sorting lists according to number of downloads
+                    for subcomponent, entry in advisory_items.items():
-		print ('[+] Sorting according to number of downloads...')
+                        vulnerability_items = {}
-		sorted_experimental = sorted(experimental.items(), key=lambda x: int(x[1]), reverse=True)
+                        vulnerability_type = re.findall('(Vulnerability Type:\s?(.*?)<)', entry[0])
-		sorted_alpha = sorted(alpha.items(), key=lambda x: int(x[1]), reverse=True)
+                        vulnerability_type.reverse()
-		sorted_beta = sorted(beta.items(), key=lambda x: int(x[1]), reverse=True)
+                        for type_entry in vulnerability_type:
-		sorted_stable = sorted(stable.items(), key=lambda x: int(x[1]), reverse=True)
+                            index = entry[0].rfind(type_entry[0])
-		sorted_outdated = sorted(outdated.items(), key=lambda x: int(x[1]), reverse=True)
+                            vulnerability_items[type_entry[1]] = []
-		sorted_allExt = sorted(allExt.items(), key=lambda x: int(x[1]), reverse=True)
+                            vulnerability_items[type_entry[1]].append(entry[0][index:])
+                            entry[0] = entry[0][:index]
 
-		print ('[+] Generating files...')
+                        for vuln_type, vuln_description in vulnerability_items.items():
-		f = open(os.path.join(self.__path, 'extensions', 'experimental_extensions'), 'w')
+                            cve = re.search(':\s?(CVE-.*?)(<|\"|\()', vuln_description[0])
-		for i in range(0,len(sorted_experimental)):
+                            if cve:
-			f.write(sorted_experimental[i][0]+'\n')
+                                cve = cve.group(1)
-		f.close()
+                            else:
+                                cve = 'None assigned'
+                            search_affected = re.search('Affected Version[s]?:\s?(.+?)<', vuln_description[0])
+                            if search_affected:
+                                affected_versions = search_affected.group(1)
+                            else:
+                                affected_versions = re.search('Affected Version[s]?:\s?(.+?)<', beauty_html).group(1)
+                            # separate versions
+                            affected_versions = affected_versions.replace("and below", " - 0.0.0")
+                            affected_versions = affected_versions.replace(";", ",")
+                            affected_versions = affected_versions.replace(' and', ',')
+                            versions = affected_versions.split(', ')
+                            for version in versions:
+                                version = re.findall('([0-9]+\.[0-9x]+\.?[0-9x]?[0-9x]?)', version)
+                                if len(version) == 0:
+                                    print("[!] Unknown version info! Skipping...")
+                                    print(" \u251c Advisory:", advisory)
+                                    print(" \u251c Subcomponent:", subcomponent)
+                                    print(" \u251c Vulnerability:", vuln_type)
+                                    print(" \u251c Versions:", affected_versions)
+                                    break
+                                elif len(version) == 1:
+                                    version = version[0]
+                                    if len(version) == 3: # e.g. version 6.2
+                                        version = version + '.0'
+                                    affected_version_max = version
+                                    affected_version_min = version
+                                else:
+                                    if parse_version(version[0]) >= parse_version(version[1]):
+                                        affected_version_max = version[0]
+                                        affected_version_min = version[1]
+                                    else:
+                                        affected_version_max = version[1]
+                                        affected_version_min = version[0]
+                                # add vulnerability
+                                vulnerabilities.append([advisory, vuln_type, subcomponent, affected_version_max, affected_version_min, cve])
+                except Exception as e:
+                    print("Error on receiving data for https://typo3.org/security/security-advisory/{}".format(advisory))
+                    print(e)
+                    exit(-1)
 
-		f = open(os.path.join(self.__path, 'extensions', 'alpha_extensions'), 'w')
+                # Add vulnerability details to database
-		for i in range(0,len(sorted_alpha)):
+                for ext_vuln in vulnerabilities:
-			f.write(sorted_alpha[i][0]+'\n')
+                    c.execute('SELECT * FROM core_vulns WHERE advisory=? AND vulnerability=? AND subcomponent=? AND affected_version_max=? AND affected_version_min=? AND cve=?', (ext_vuln[0], ext_vuln[1], ext_vuln[2], ext_vuln[3], ext_vuln[4], ext_vuln[5],))
-		f.close()
+                    data = c.fetchall()
+                    if not data:
+                        update_counter+=1
+                        c.execute('INSERT INTO core_vulns VALUES (?,?,?,?,?,?)', (ext_vuln[0], ext_vuln[1], ext_vuln[2], ext_vuln[3], ext_vuln[4], ext_vuln[5],))
+                        conn.commit()
+                    else:
+                        if update_counter == 0:
+                            print('[!] Already up-to-date.\n')
+                        else:
+                            print('[+] Done.')
+                            print('[!] Added {} new CORE vulnerabilities to database.\n'.format(update_counter))
+                        return True
 
-		f = open(os.path.join(self.__path, 'extensions', 'beta_extensions'),'w')
+    def dlProgress(self, count, blockSize, totalSize):
-		for i in range(0,len(sorted_beta)):
+        """
-			f.write(sorted_beta[i][0]+'\n')
+            Progressbar for extension download
-		f.close()
+        """
+        percent = int(count*blockSize*100/totalSize)
+        sys.stdout.write('\r \u251c Downloading ' + '%d%%' % percent)
+        sys.stdout.flush()
 
-		f = open(os.path.join(self.__path, 'extensions', 'stable_extensions'), 'w')
+    def download_ext(self):
-		for i in range(0,len(sorted_stable)):
+        """
-			f.write(sorted_stable[i][0]+'\n')
+            Download extensions from server and unpack the ZIP
-		f.close()
+        """
+        print('[+] Getting extension file...')
+        try:
+            # Maybe someday we need to use mirrors: https://repositories.typo3.org/mirrors.xml.gz
+            urllib.request.urlretrieve('https://typo3.org/fileadmin/ter/extensions.xml.gz', 'extensions.xml.gz', reporthook=self.dlProgress)
+            with gzip.open('extensions.xml.gz', 'rb') as infile:
+                with open('extensions.xml', 'wb') as outfile:
+                    for line in infile:
+                        outfile.write(line)
+            infile.close()
+            outfile.close()
+        except Exception as e:
+            print ('\n', e)
 
-		f = open(os.path.join(self.__path, 'extensions', 'outdated_extensions'), 'w')
+    def load_extensions(self):
-		for i in range(0,len(sorted_outdated)):
+        """
-			f.write(sorted_outdated[i][0]+'\n')
+            Parse the extension file and add extensions in database
-		f.close()
+        """
+        print('\n \u251c Parsing extension file...')
+        tree = ElementTree.parse('extensions.xml') 
+        root = tree.getroot()
 
-		f = open(os.path.join(self.__path, 'extensions', 'all_extensions'), 'w')
+        # for every extension get:
-		for i in range(0,len(sorted_allExt)):
+        #     title, extensionkey, description, version, state
-			f.write(sorted_allExt[i][0]+'\n')
+        for extensions in root:
-		f.close()
+            title = extensions[1][0].text
+            extensionkey = extensions.get('extensionkey')
+            description = extensions[1][1].text
+            version = '0.0.0'
+            state = ''
 
-		print ('[+] Loaded', len(sorted_allExt), 'extensions')
+            # search for current version
-		os.remove('extensions.xml.gz')
+            for extension in extensions.iter('version'):
-		os.remove('extensions.xml')
+                if not(extension.attrib['version'] == ''):
+                    try:
+                        if parse_version((extension.attrib['version']).split('-')[0]) > parse_version(version):
+                            version = extension.attrib['version']
+                            state = (extension.find('state')).text
+                    except ValueError:
+                        pass
+            c.execute('INSERT OR REPLACE INTO extensions VALUES (?,?,?,?,?)', (title, extensionkey, description, version, state))
+
+        conn.commit()
+        os.remove('extensions.xml.gz')
+        os.remove('extensions.xml')
+        print(' \u2514 Done. Added {} extensions to database'.format(len(root.findall('extension'))))
+
+    def load_extension_vulns(self):
+        """
+            Grep the EXTENSION vulnerabilities from the security advisory website
+            
+            Search for advisories and maximum pages
+            Request every advisory and get:
+                Advisory Title
+                Extension Name
+                Vulnerability Type
+                Affected Versions
+        """
+        print('\n[+] Searching for new extension vulnerabilities...')
+        update_counter = 0
+        response = requests.get('https://typo3.org/help/security-advisories/typo3-extensions/1')
+        pages = re.findall('<a class=\"page-link\" href=\"/help/security-advisories/typo3-extensions/([0-9]+)\">', response.text)
+        last_page = int(pages[-1])
+
+        for current_page in range(1, last_page+1):
+            print(' \u251c Page {}/{}'.format(current_page, last_page))
+            response = requests.get('https://typo3.org/help/security-advisories/typo3-extensions/{}'.format(current_page), timeout=6)
+            advisories = re.findall('TYPO3-EXT-SA-[0-9][0-9][0-9][0-9]-[0-9][0-9][0-9]', response.text)
+            for advisory in advisories:
+                vulnerabilities = []
+                affected_version_max = '0.0.0' 
+                affected_version_min = '0.0.0'
+                # adding vulns with odd stuff on website
+                if advisory == 'TYPO3-EXT-SA-2014-018':
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-018', 'phpmyadmin', 'Cross-Site Scripting, Denial of Service, Local File Inclusion', '4.18.4', '4.18.0'])
+                elif advisory == 'TYPO3-EXT-SA-2014-015':
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-015', 'dce', 'Information Disclosure', '0.11.4', '0.0.0'])
+                elif advisory == 'TYPO3-EXT-SA-2014-013':
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-013', 'cal', 'Denial of Service', '1.5.8', '0.0.0'])
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-013', 'cal', 'Denial of Service', '1.6.0', '1.6.0'])
+                elif advisory == 'TYPO3-EXT-SA-2014-009':
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-009', 'news', 'Cross-Site Scripting', '3.0.0', '3.0.0'])
+                    vulnerabilities.append(['TYPO3-EXT-SA-2014-009', 'news', 'Cross-Site Scripting', '2.3.0', '2.0.0'])
+                else:
+                    try:
+                        html = requests.get('https://typo3.org/security/advisory/{}'.format(advisory.lower()))
+                        beauty_html = html.text.replace('\xa0', ' ')
+                        beauty_html = beauty_html.replace('</strong>', '')
+                        beauty_html = beauty_html.replace('&nbsp;', ' ')
+                        beauty_html = beauty_html.replace('&amp;', '&')
+                        advisory_info = re.search('<title>(.*)</title>', beauty_html).group(1)
+                        vulnerability = re.findall('Vulnerability Type[s]?:\s?(.*?)<', beauty_html)
+                        affected_versions = re.findall('Affected Version[s]?:\s?(.+?)<', beauty_html)
+                        extensionkey = re.findall('Extension[s]?:\s?(.*?)<', beauty_html)
+                        # Sometimes there are multiple extensions in an advisory
+                        if len(extensionkey) == 0: # If only one extension affected
+                            extensionkey = [advisory_info[advisory_info.find('('):]]
+                        for item in range (0, len(extensionkey)):
+                            extensionkey_item = extensionkey[item]
+                            extensionkey_item = extensionkey_item[extensionkey_item.rfind('(')+1:extensionkey_item.rfind(')')]
+                            description = vulnerability[item]
+                            version_item = affected_versions[item]
+                            version_item = version_item.replace("and all versions below", "- 0.0.0")
+                            version_item = version_item.replace("and all version below", "- 0.0.0") # typo
+                            version_item = version_item.replace("and alll versions below", "- 0.0.0") # typo
+                            version_item = version_item.replace("and below of", "-")
+                            version_item = version_item.replace("and below", "- 0.0.0")
+                            version_item = version_item.replace("&nbsp;", " ")
+                            version_item = version_item.replace(";", ",")
+                            version_item = version_item.replace(' and', ',')
+                            versions = version_item.split(', ')
+                            for version in versions:
+                                version = re.findall('([0-9]+\.[0-9x]+\.[0-9x]+)', version)
+                                if len(version) == 1:
+                                    affected_version_max = version[0]
+                                    affected_version_min = version[0]
+                                else:
+                                    if parse_version(version[0]) >= parse_version(version[1]):
+                                        affected_version_max = version[0]
+                                        affected_version_min = version[1]
+                                    else:
+                                        affected_version_max = version[1]
+                                        affected_version_min = version[0]
+                                vulnerabilities.append([advisory, extensionkey_item, description, affected_version_max, affected_version_min])
+                    except Exception as e:
+                        print("Error on receiving data for https://typo3.org/security/advisory/{}".format(advisory))
+                        print(e)
+                        exit(-1)
+
+                # Add vulnerability details to database
+                for ext_vuln in vulnerabilities:
+                    c.execute('SELECT * FROM extension_vulns WHERE advisory=? AND extensionkey=? AND vulnerability=? AND affected_version_max=? AND affected_version_min=?', (ext_vuln[0], ext_vuln[1], ext_vuln[2], ext_vuln[3], ext_vuln[4],))
+                    data = c.fetchall()
+                    if not data:
+                        update_counter+=1
+                        c.execute('INSERT INTO extension_vulns VALUES (?,?,?,?,?)', (ext_vuln[0], ext_vuln[1], ext_vuln[2], ext_vuln[3], ext_vuln[4]))
+                        conn.commit()
+                    else:
+                        if update_counter == 0:
+                            print('[!] Already up-to-date.\n')
+                        else:
+                            print(' \u2514 Done. Added {} new EXTENSION vulnerabilities to database.\n'.format(update_counter))
+                        return True

lib/version_information.py

@@ -1,55 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-#-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
-#-------------------------------------------------------------------------------
-
-import re
-import time
-from lib.request import Request
-
-class VersionInformation:
-	"""
-	This class will search for version information.
-		The exact version can be found in the ChangeLog, therefore it will be requested first.
-		Less specific version information can be found in the NEWS or INSTALL file. 
-	"""
-	def search_typo3_version(self, domain):
-		files = {'/typo3_src/ChangeLog':'[Tt][Yy][Pp][Oo]3 (\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',  
-				'/ChangeLog':'[Tt][Yy][Pp][Oo]3 (\d{1,2}\.\d{1,2}\.?[0-9]?[0-9]?)',
-				'/typo3_src/NEWS.txt':'http://wiki.typo3.org/TYPO3_(\d{1,2}\.\d{1,2})', 
-				'/typo3_src/NEWS.md':'[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}\.\d{1,2}) - WHAT\'S NEW', 
-				'/NEWS.txt':'http://wiki.typo3.org/TYPO3_(\d{1,2}\.\d{1,2})', 
-				'/NEWS.md':'[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}\.\d{1,2}) - WHAT\'S NEW',
-				'/INSTALL.md':'[Tt][Yy][Pp][Oo]3 [Cc][Mm][Ss] (\d{1,2}(.\d{1,2})?)',
-				'/INSTALL.md':'[Tt][Yy][Pp][Oo]3 v(\d{1})'
-				}
-
-		version = 'could not be determined'
-		for path, regex in files.items():
-			response = Request.version_information(domain.get_name(), path, regex)
-
-			if not (response is None):
-				string = '[!] ' + 'Found version file:'
-				print(string.ljust(30) + path)
-
-				if (version is 'could not be determined'):
-					version = response
-				elif (len(response) > len(version)):
-					version = response
-
-		domain.set_typo3_version(version)

requirements.txt

@@ -0,0 +1,3 @@
+requests
+colorama
+progressbar

typo3_enumerator.py

@@ -1,195 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-#-------------------------------------------------------------------------------
-# Typo3 Enumerator - Automatic Typo3 Enumeration Tool
-# Copyright (c) 2014-2017 Jan Rude
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
-#-------------------------------------------------------------------------------
-
-__version__ = '0.4.5.2'
-__program__ = 'Typo-Enumerator'
-__description__ = 'Automatic Typo3 enumeration tool'
-__author__ = 'https://github.com/whoot'
-
-import sys
-import os.path
-import datetime
-import argparse
-import json
-import inspect
-import base64
-from colorama import Fore, init, deinit, Style
-from lib.check_installation import Typo3_Installation
-from lib.version_information import VersionInformation
-from lib.extensions import Extensions
-from lib.domain import Domain
-from lib.update import Update
-from lib.output import Output
-init()
-
-class Typo3:
-	def __init__(self):
-		self.__domain_list = []
-		self.__extensions = None
-		self.__path = path = os.path.dirname(os.path.abspath(__file__))
-
-	def print_help():
-		print(
-"""\nUsage: python3 typo3_enumerator.py [options]
-
-Options:
-  -h, --help 		Show this help message and exit
-
-  Target:
-   At least one of these options has to be provided to define the target(s)
-
-    -d [DOMAIN, ...], --domain [DOMAIN, ...] Target domain(s)
-    -f FILE, --file FILE 		     Parse targets from file (one domain per line)
-
-
-  Optional:
-   You dont need to specify this arguments, but you may want to
-
-    --top TOP 		Test if top [TOP] downloaded extensions are installed
-			  Default: every in list
-    --state STATE 	Extension state [all, experimental, alpha, beta, stable, outdated]
-			  Default: all
-    --timeout TIMEOUT 	The timeout for all requests
-			  Default: 10 seconds
-    --agent USER_AGENT 	The user-agent used for all requests
-			  Default: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
-    --auth USER:PASS 	Username and Password for HTTP Basic Authorization
-			  Default: admin:admin
-    --cookie COOKIE     Cookie
-    --threads THREADS 	The number of threads used for enumerating the extensions
-			  Default: 5
-
-
-  Anonymity:
-   This options can be used to proxy all requests through TOR/Privoxy
-
-    --tor 	    	Using only TOR for connections
-    --port PORT     	Port for TOR
-    			  Default: 9050
-
-  General:
-    -u, --update        Update TYPO3 extensions
-""")
-
-	def run(self):
-		parser = argparse.ArgumentParser(add_help=False)
-		group = parser.add_mutually_exclusive_group()
-		anonGroup = parser.add_mutually_exclusive_group()
-		help = parser.add_mutually_exclusive_group()
-		group.add_argument('-f', '--file', dest='file')
-		group.add_argument('-d', '--domain', dest='domain', type=str, nargs='+')
-		group.add_argument('-u', '--update', dest='update', action='store_true')
-		parser.add_argument('--top', type=int, dest='top', metavar='VALUE')
-		parser.add_argument('--state', dest='ext_state', choices = ['all', 'experimental', 'alpha', 'beta', 'stable', 'outdated'], nargs='+', default = ['all'])
-		anonGroup.add_argument('--tor', action='store_true')
-		parser.add_argument('-p', '--port', dest='port', type=int)
-		parser.add_argument('--threads', dest='threads', type=int, default = 5)
-		parser.add_argument('--agent', dest='agent', type=str, default = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0')
-		parser.add_argument('--auth', dest='auth', type=str, default = 'admin:admin')
-		parser.add_argument('--cookie', dest='cookie', type=str, default = 'typo3enum=none')
-		parser.add_argument('--timeout', dest='timeout', type=int, default = 10)
-		help.add_argument( '-h', '--help', action='store_true')
-		args = parser.parse_args()
-
-		if args.help:
-			Typo3.print_help()
-			return True
-
-		try:
-			if args.update:
-				Update(self.__path)
-				return True
-
-			if args.tor:
-				from lib.tor import Tor
-				if args.port:
-					tor = Tor(args.port)
-				else:
-					tor = Tor()
-				tor.start_daemon()
-				tor.connect()
-
-			if args.domain:
-				for dom in args.domain:
-					self.__domain_list.append(Domain(dom, args.ext_state, args.top))
-			elif args.file:
-				if not os.path.isfile(args.file):
-					print(Fore.RED + '\n[x] File not found: ' + args.file + '\n |  Aborting...' + Fore.RESET)
-					sys.exit(-1)
-				else:
-					with open(args.file, 'r') as f:
-						for line in f:
-							self.__domain_list.append(Domain(line.strip('\n'), args.ext_state, args.top))
-
-			config = {'threads':args.threads, 'agent':args.agent, 'timeout':args.timeout, 'cookie':args.cookie, 'user': (args.auth).split(':')[0], 'pass': (args.auth).split(':')[1]}
-			json.dump(config, open(os.path.join(self.__path, 'lib', 'config.json'), 'w'))
-
-			for domain in self.__domain_list:
-				print('\n\n' + Fore.CYAN + Style.BRIGHT + '[ Checking ' + domain.get_name() + ' ]' + '\n' + '-'* 73  + Fore.RESET + Style.RESET_ALL)
-				Typo3_Installation.run(domain)
-				for key, value in domain.get_interesting_headers().items():
-					Output.interesting_headers(key, value)
-				if not domain.get_typo3():
-					print(Fore.RED + '\n[x] It seems that Typo3 is not used on this domain' + Fore.RESET)
-				else:
-					version = VersionInformation()
-					version.search_typo3_version(domain)
-					login = Typo3_Installation.search_login(domain)
-					Output.typo3_installation(domain)
-					# Loading extensions
-					if (self.__extensions is None):
-						ext = Extensions(args.ext_state, args.top, self.__path)
-						self.__extensions = ext.load_extensions()
-					# copy them in domain object
-					if (domain.get_extensions() is None):
-						domain.set_extensions(self.__extensions)
-					# search
-					print ('\n[ Searching', len(self.__extensions), 'extensions ]')
-					ext.search_extension(domain, self.__extensions)
-					ext.search_ext_version(domain, domain.get_installed_extensions())
-					Output.extension_output(domain.get_path(), domain.get_installed_extensions())
-		
-		except KeyboardInterrupt:
-			print('\nReceived keyboard interrupt.\nQuitting...')
-			exit(-1)
-		finally:
-			deinit()
-			now = datetime.datetime.now()
-			print('\n\n' + __program__ + ' finished at ' + now.strftime('%Y-%m-%d %H:%M:%S') + '\n')
-		
-if __name__ == '__main__':
-	print('\n' + 73*'=' + Style.BRIGHT)
-	print(Fore.BLUE)
-	print(' _______                     ______ '.center(73))
-	print('|_     _|.--.--.-----.-----.|__    |'.center(73))
-	print('  |   |  |  |  |  _  |  _  ||__    |'.center(73))
-	print('  |___|  |___  |   __|_____||______|'.center(73))
-	print('         |_____|__|                 '.center(73))
-	print(' _______                                         __              '.center(73))
-	print('|    ___|.-----.--.--.--------.-----.----.---.-.|  |_.-----.----.'.center(73)) 
-	print('|    ___||     |  |  |        |  -__|   _|  _  ||   _|  _  |   _|'.center(73))
-	print('|_______||__|__|_____|__|__|__|_____|__| |___._||____|_____|__|  '.center(73))
-	print(Fore.RESET + Style.RESET_ALL)
-	print(__description__.center(73))
-	print(('Version ' + __version__).center(73))
-	print((__author__).center(73))
-	print(73*'=')
-	main = Typo3()
-	main.run()

typo3scan.py

@@ -0,0 +1,193 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#-------------------------------------------------------------------------------
+# Typo3Scan - Automatic Typo3 Enumeration Tool
+# Copyright (c) 2014-2020 Jan Rude
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses/)
+#-------------------------------------------------------------------------------
+
+__version__ = '0.5'
+__program__ = 'Typo3Scan'
+__description__ = 'Automatic Typo3 enumeration tool'
+__author__ = 'https://github.com/whoot'
+
+import sys
+import json
+import sqlite3
+import os.path
+import argparse
+from lib.domain import Domain
+from lib.update import Update
+from lib.initdb import DB_Init
+from lib.extensions import Extensions
+from colorama import Fore, init, deinit, Style
+init()
+
+class Typo3:
+    def __init__(self):
+        self.__domain_list = []
+        self.__path = os.path.dirname(os.path.abspath(__file__))
+        self.__extensions = []
+
+    def print_help():
+        print(
+"""\nUsage: python typo3scan.py [options]
+
+Options:
+  -h, --help         Show this help message and exit.
+
+  Target:
+   At least one of these options has to be provided to define the target(s):
+
+    --domain | -d <target url>  The Typo3 URL(s)/domain(s) to scan.
+    --file   | -f <file>        Parse targets from file (one domain per line).
+
+
+  Optional:
+   You dont need to specify this arguments, but you may want to
+
+    --vuln              Check for extensions with known vulnerabilities only.
+                        Default: all
+              
+    --timeout TIMEOUT   Request Timeout.
+                        Default: 10 seconds
+              
+    --auth USER:PASS    Username and Password for HTTP Basic Authorization.
+    
+    --cookie NAME=VALUE Can be used for authenticiation based on cookies.
+         
+    --threads THREADS   The number of threads to use for enumerating extensions.
+                        Default: 5
+
+  General:
+    -u | --update       Update the database.
+    -r | --reset        Reset the database.
+""")
+
+    def run(self):
+        parser = argparse.ArgumentParser(add_help=False)
+        group = parser.add_mutually_exclusive_group()
+        help = parser.add_mutually_exclusive_group()
+        group.add_argument('-f', '--file', dest='file')
+        group.add_argument('-d', '--domain', dest='domain', type=str, nargs='+')
+        group.add_argument('-u', '--update', dest='update', action='store_true')
+        group.add_argument('-r', '--reset', dest='reset', action='store_true')
+        parser.add_argument('--vuln', dest='vuln', action='store_true')
+        parser.add_argument('--threads', dest='threads', type=int, default=5)
+        parser.add_argument('--auth', dest='auth', type=str, default='')
+        parser.add_argument('--cookie', dest='cookie', type=str, default='')
+        parser.add_argument('--timeout', dest='timeout', type=int, default=10)
+        help.add_argument( '-h', '--help', action='store_true')
+        args = parser.parse_args()
+
+        if args.help or not len(sys.argv) > 1:
+            Typo3.print_help()
+
+        elif args.reset:
+            DB_Init()
+
+        elif args.update:
+            Update()
+
+        else:
+            database = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'lib', 'typo3scan.db')
+            conn = sqlite3.connect(database)
+            c = conn.cursor()       
+            c.execute('SELECT * FROM UserAgents ORDER BY RANDOM() LIMIT 1;')
+            user_agent = c.fetchone()[0]
+            c.close()
+            config = {'threads': args.threads, 'timeout': args.timeout, 'cookie': args.cookie, 'auth': args.auth, 'User-Agent': user_agent}
+            json.dump(config, open(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'lib', 'config.json'), 'w'))
+            try:
+                if args.domain:
+                    for dom in args.domain:
+                        self.__domain_list.append(dom)
+                elif args.file:
+                    if not os.path.isfile(args.file):
+                        print(Fore.RED + '\n[x] File not found: {}\n |  Aborting...'.format(args.file) + Fore.RESET)
+                        sys.exit(-1)
+                    else:
+                        with open(args.file, 'r') as f:
+                            for line in f:
+                                self.__domain_list.append(line.strip())
+
+                for domain in self.__domain_list:
+                    print(Fore.CYAN + Style.BRIGHT + '\n\n[ Checking {} ]\n'.format(domain) + '-'* 73  + Fore.RESET + Style.RESET_ALL)
+                    check = Domain(domain)
+                    check.check_root()
+                    default_files = check.check_default_files()
+                    if not default_files:
+                        check_404 = check.check_404()
+                    if not check.is_typo3():
+                        print(Fore.RED + '\n[x] It seems that Typo3 is not used on this domain' + Fore.RESET)
+                    else:
+                        # print interesting headers
+                        print('\n[+] Interesting Headers')
+                        for key, value in check.get_interesting_headers().items():
+                            string = ' \u251c {}:'.format(key)
+                            print(string.ljust(30) + value)
+
+                        # check for typo3 information
+                        print('\n[+] Typo3 Information')
+                        check.search_login()
+                        check.search_typo3_version()
+
+                        # Search extensions
+                        print(' [+] Extension Search')
+                        if not self.__extensions:
+                            conn = sqlite3.connect(database)
+                            c = conn.cursor()
+                            if args.vuln:
+                                for row in c.execute('SELECT extensionkey FROM extension_vulns'):
+                                    self.__extensions.append(row[0])
+                                self.__extensions = set(self.__extensions)
+                            else:
+                                for row in c.execute('SELECT extensionkey FROM extensions'):
+                                    self.__extensions.append(row[0])
+                            conn.close()
+                        print ('  \u251c Brute-Forcing {} extensions'.format(len(self.__extensions)))
+                        extensions = Extensions()
+                        ext_list = extensions.search_extension(check.get_path(), self.__extensions, args.threads)
+                        if ext_list:
+                            print ('\n  \u251c Found {} extensions'.format(len(ext_list)))
+                            print ('  |\n  \u251c Brute-Forcing version information'.format(len(self.__extensions)))
+                            ext_list = extensions.search_ext_version(ext_list, args.threads)
+                            extensions.output(ext_list, database)
+                        else:
+                            print ('\n  [!] No extensions found.')
+            except KeyboardInterrupt:
+                print('\nReceived keyboard interrupt.\nQuitting...')
+                exit(-1)
+            finally:
+                deinit()
+                print()
+        
+if __name__ == '__main__':
+    print('\n' + 73*'=' + Style.BRIGHT)
+    print(Fore.CYAN)
+    print('________                   ________   _________                     '.center(73))
+    print('\_    _/__ __ ______  _____\_____  \ /   _____/ ____ _____    ___  '.center(73))
+    print('  |  | |  |  |\____ \|  _  | _(__  < \_____  \_/ ___\\\\__  \  /   \ '.center(73))
+    print('  |  | |___  ||  |_) | (_) |/       \/        \  \___ / __ \|  |  \ '.center(73))
+    print('  |__| / ____||   __/|_____|________/_________/\_____|_____/|__|__/ '.center(73))
+    print('       \/     |__|                                                  '.center(73))
+    print(Fore.RESET + Style.RESET_ALL)
+    print(__description__.center(73))
+    print(('Version ' + __version__).center(73))
+    print((__author__).center(73))
+    print(73*'=')    
+
+    main = Typo3()
+    main.run()