Sometimes, one wants to see whats going on on an embedded device, or maybe, what might have been going on in the past. For reliability and stability reasons, this is often not possibe as the UI provided by a device is there to protect the user from themselves. Yes, they might know what their doing, but it will break during the next update! In return, the limitations resulting from a specific UI, make deeper insights harder. Luckily, Fortigate firewalls allow certain deeper going access. Here an inisight into what is possible, how, and a little script to help, when scraping potential IoCs from the devices, or just comparing them to a known good state!
Backend Shell
Fortigates “allow” direct shell access as described
here
. Sadly my first test Fortigate,a Fortigate 80C running FortiOS 5.4.10 at home was a little bit too old, so I needed to work with fnsysctl. Not cool, but hey…
So i.e. fnsysctl ls /bin will list the contents of /bin …
FGT80C3911612795 # fnsysctl ls -all /bin
drwxr-xr-x 2 0 0 Wed Apr 16 01:19:28 2025 2940 .
drwxr-xr-x 13 0 0 Wed Apr 16 01:19:27 2025 320 ..
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 acd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 adsl_mon -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 alarmd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 alertmail -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 authd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 bgpd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 cardctl -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 cardmgr -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 chat -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 chlbd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 11 cli_grep -> /bin/sysctl
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 cmdbsvr -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 confsyncd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 confsynchbd -> /bin/init
lrwxrwxrwx 1 0 0 Wed Apr 16 01:19:23 2025 9 csfd -> /bin/init
In addition, my FortiOS 5.4.10 does not have any command, which allows the calculation of checksums, thus my extraction script utilizes fnsysctl in combination with ls -l and cat and then does the rest on my box. This combination is automated using
paramiko
as an SSH client in python and a plain recursive function, iterating through all folders and turning the structure into dict and then storing it as json. While doing so, each file is “copied” to my box using cat, MD5, SHA1 and SHA256 sums calculated in RAM and, if needed, a copy locally stored for further analysis. Symlinks are also documented as such.
For me, initially, the hashes were the crucial aspect, as they allowed me to compare the content on two different Fortigates, thus the low-tech approach of an integrity check. Having read
Fortinets documentation of a threat actor
changing symlinks, this was my second focus.
FortiOS 7.2.6
In addition to the previously used Fortigate 80C, I just recevied a Fortigate 60E running FortiOS 7.2.6. Here a few things have changed, as also mentioned in the
Fortigate docs
. While fnsysctl ls works like a charm, using fnsysctl cat results in a Not allowed error. Following the instructions, I tried to enable shell-access in config system globel via SSH and afterwards also via the webinterface’s CLI, but …
FortiGate-60E # config system global
FortiGate-60E (global) # set shell-access enable
command parse error before 'shell-access'
Command fail. Return code -61
And so far, I have not been able to activate the backend shell, but luckily I didn’t need it to proceed.
Hashes
My 60E supports an internal command to generate file hashes on the Fortigate
as documented here
. As it seems diagnose sys filesystem hash will iterate through all folders from its basedir and print the sha256 hashes of all files it finds.
FortiGate-60E # FortiGate-60E # diagnose sys filesystem hash
SYNOPSIS
diagnose sys fshash [OPTION...] [PATH...]
DESCRIPTION
Compute sha256 hash for each file in the directory specified by each PATH.
OPTIONS
-d [depth]
Specify maximum depth of traversal.
diagnose sys fshash does not seem to function, guess it might be the commands initial syntax name.
And running the command…
FortiGate-60E # diagnose sys filesystem hash
Hash contents: /bin
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/syslogd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/acd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/httpsnifferd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/merged_daemons -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/sdncd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/forticldd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/ipamd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/pimd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/cid -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/memuploadd -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/lldptx -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/fsso_ldap -> /bin/init
110d87008eb08d0a2dff5a1b5b24baa43854a8ca5eac2f8a95a9871373c0deae /bin/ntpd -> /bin/init
.
.
.
.
f3da3a2317d2601a5d30de6a3076de34352e21041e0cd0e17a86b3bba2bf40ee /node-scripts/0007d691b3a7519f20fa76f56e6e2fc0.node
Hash contents: /sbin
d1655b44ae15dfd76e487106b618bb7cdde5a1c13ef6710a909363912a72b9da /sbin/init
Hash contents: /usr/local
a94204febf25b4875c4ca7724757f15bfc6b1b587c21e31f321f29ada3224316 /usr/local/apache2/modules/mod_watchdog.so
6dd02060d9db11dd36a303bfa822eb0d66a72e221ce6be9b1e6919a335b50dd6 /usr/local/apache2/modules/mod_md.so
f8d2278fb5f0b5bc6f11a74ad1213ace04c185a883df0ab5c171bbc26281da23 /usr/local/apache2/conf/md-acme.conf
b3f5856b10d11440535f07fee363bf77ec5986ab1efd5a50c3206113b1aa5881 /usr/local/apache2/conf/httpd.conf
63e3f86bd88e22a856774026e13d67f7e564364775997f83fb96e5610d919bf0 /usr/local/apache2/conf/admin-vhost.conf
616a1362e10e059922e19072fba1bd1ce0abf0fed6ad58b1a950d999a7279761 /usr/local/apache2/conf/admin-global.conf
ce95e59e1f7fed0ebda42b61de49d213f85ed31b73214cccf08b47cb98f81814 /usr/local/apache2/conf/mime.types
Filesystem hash complete. Hashed 1928 files.
It prints all hashes in addition to the paths of each file. So I could have saved a lot of time starting with a newer Fortigate :)
Doing the comparison here is simple.
Conclusion
By adding diagnose sys filesystem hash, Fortinet has added a simple but effective measure to monitor the integrity of the Fortigate. Regularly extracting the hashes and comparing them with a known good, or the last state should make identifying deeper going attacks by far easier.
Notes
- paramiko does not support empty ssh passwords , usually not an issue, except for maybe in testing environments
- Modern SSH is unhappy with passwords, thus using one has to be enforced:
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no- The same applies to
paramiko:client.connect(host, username=username, password=password, allow_agent=False,look_for_keys=False)
- The same applies to
- The JSON generated from my Fortigate 80C can be found here
- All hashes from my Fortigate 60E can be found here
Script
The following script implements the fetch mechanism utilizing fnsysctl and cat, thus the method for old Fortigates.
Usage
python3 fortifori.py -v fetch admin 1234 192.168.130.152 output.json can be used to fetch data from a Fortigate. To compare two datasets python3 fortifori.py -v verify old.json output.json will help you.
Python
import paramiko
import hashlib
import json
import os
import argparse
import base64
import datetime
import sys
start_time = None
path_blacklist = ['/dev', '/proc']
#print("!!!!! WARNING !!!!!\n This script will not search in the following paths\n" + str(path_blacklist) + "\n!!!!! WARNING !!!!!\n")
def process_file(path, client, console_string_b, verbose):
_stdin, _stdout,_stderr = client.exec_command("fnsysctl cat " + path)
t = _stdout.read()
t = t.replace(b'\n' + console_string_b,b'')
t = t.replace(console_string_b,b'')
result = {}
result['type'] = "file"
result['md5'] = hashlib.md5(t).hexdigest()
result['sha1'] = hashlib.sha1(t).hexdigest()
result['sha256'] = hashlib.sha256(t).hexdigest()
if len(t) > 100:
result['head'] = base64.b64encode(t[:100]).decode()
else:
result['head'] = base64.b64encode(t[:len(t)]).decode()
return result
def make_timestamp(data):
return datetime.datetime.strptime(data,'%Y%b%d %H:%M:%S').strftime('%Y-%m-%d %H:%M:%S')
def parse_folder(path, client, console_string_b, verbose):
if verbose:
print("Working on Folder " + path)
result = {}
result[path] = {}
result[path]['type'] = "folder"
result[path]['contents'] = {}
blacklisted = False
for p in path_blacklist:
if path.startswith(p):
blacklisted = True
break
if not blacklisted:
_stdin, _stdout,_stderr = client.exec_command("fnsysctl ls -l " + path)
t = _stdout.read()
t = t.replace(b'\n' + console_string_b,b'')
t = t.replace(console_string_b,b'')
data = t.split(b'\n')
for d in data:
d = d.split()
if len(d) > 1:
if not b'd' in d[0]:
if len(d) == 13:
result[path]['contents'][d[10].decode()] = {'type' : 'symlink', 'target' : d[12].decode()}
result[path]['contents'][d[10].decode()]['date'] = make_timestamp(d[8].decode() +d[5].decode() + d[6].decode() + " " +d[7].decode())
else:
result[path]['contents'][d[10].decode()] = process_file(os.path.join(path,d[10].decode()), client, console_string_b, verbose)
result[path]['contents'][d[10].decode()]['size'] = d[9].decode()
result[path]['contents'][d[10].decode()]['date'] = make_timestamp(d[8].decode() + d[5].decode() + d[6].decode() + " " +d[7].decode())
else:
tf = parse_folder(os.path.join(path,d[10].decode()), client, console_string_b, verbose)
result[path]['contents'].update(tf)
else:
result[path]['comment'] = 'blacklisted'
return result
def get_console_string(client, verbose):
if verbose:
print("Fetching console string\n")
_stdin, _stdout,_stderr = client.exec_command("fnsysctl")
t = _stdout.read()
#Expecting something like
#~~
#FGT80C3911612795 # fnsysctl
#
#FGT80C3911612795 #
#~~
t = t.split(b'\n')
if len(t) == 2 and t[0] == t[1]:
if verbose:
print("Extracted \"" + t[1].decode() + "\" as console string")
return t[1]
def get_dataset_with_fnsysctl(username, password, host, verbose):
client = paramiko.client.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(host, username=username, password=password, allow_agent=False,look_for_keys=False)
console_string_b = get_console_string(client, verbose)
out = parse_folder("/", client, console_string_b, verbose)
client.close()
return out
def fetch(args):
r = get_dataset_with_fnsysctl(args.username, args.password, args.host, args.verbose)
rj = json.dumps(r, indent=4, sort_keys=True)
f = open(args.target_file,'w')
f.write(rj)
f.close()
def compare_file(good, new, verbose):
result = {}
for x in good:
if not good[x] == new[x]:
result[x] = {'good': good[x], 'new':new[x]}
if len(result) > 0:
return result
else:
return False
def compare_symlink(good, new, verbose):
result = {}
for x in good:
if not good[x] == new[x]:
result[x] = {'good': good[x], 'new':new[x]}
if len(result) > 0:
return result
else:
return False
def compare_folder(good, new, verbose):
result = {}
for e in good:
if e in new:
if good[e]['type'] != new[e]['type']:
return e + " has changed it's type"
else:
if good[e]['type'] == 'file':
t = compare_file(good[e], new[e], verbose)
if t:
result[e] = t
elif good[e]['type'] == 'folder':
t = compare_folder(good[e]['contents'], new[e]['contents'], verbose)
if t:
result[e] = t
elif good[e]['type'] == 'symlink':
t = compare_symlink(good[e], new[e], verbose)
if t:
result[e] = t
else:
result[e] = e + " is missing"
for e in new:
if not e in good:
result[e] = e + " has been added"
return result
def verify(args):
good_f = open(args.known_good, 'r')
good = json.load(good_f)
good_f.close()
new_f = open(args.new, 'r')
new = json.load(new_f)
new_f.close()
result = compare_folder(good, new, args.verbose)
if len(result) == 0:
print("No differences could be identified!")
else:
print(result)
parser = argparse.ArgumentParser(prog='fortifori', description='A poor man\'s forensics tool for Fortigates', epilog='Not perfect, but it does the job!')
parser.add_argument('-v', '--verbose', help='Enable output during run (recommended)', action=argparse.BooleanOptionalAction)
subparsers = parser.add_subparsers()
parser_fetch = subparsers.add_parser('fetch', help='Fetch data set from Fortigate')
parser_fetch.add_argument('username', help='Username for SSH access to Fortigate')
parser_fetch.add_argument('password', help='Password for Fortigate User')
parser_fetch.add_argument('host', help='IP address of Fortigate')
parser_fetch.add_argument('target_file', help='File to save the result')
parser_fetch.set_defaults(func=fetch)
parser_verify = subparsers.add_parser('verify', help='Compare two configs')
parser_verify.add_argument('known_good', help='Path to known good data set')
parser_verify.add_argument('new', help='Path to new data set')
parser_verify.set_defaults(func=verify)
if len(sys.argv)==1:
parser.print_help(sys.stderr)
sys.exit(1)
args = parser.parse_args()
args.func(args)