Malware Dataset Recording from Network Traffic

v2/malware/README.rst

@@ -0,0 +1,143 @@
+=============================
+Setting up a Malware Test Bed
+=============================
+
+Here is an overview for setting up a simple malware test bed and how to configure a local, contained network for hosting this type of simulation. It is an educational guide focused on showing how the included training data was created for teaching AI how to defend against:
+
+- `Intrusion using ssh or telnet`_
+
+.. image:: https://i.imgur.com/PLzIiFR.png
+
+.. _Intrusion using ssh or telnet: https://github.com/jay-johnson/antinex-datasets/tree/v2/v2/malware/intrusion
+
+Deployment
+==========
+
+#.  Attack vms are running on one host
+#.  Defender vm is running on another host
+#.  AntiNex stack is running on a bare metal host
+
+Networking
+==========
+
+**Record and Train like it Runs on Production**
+
+Assumptions
+-----------
+
+The **intrusion malware** I am testing is in `this repo <https://github.com/jay-johnson/nerfball>`__, and it is contained inside a docker container with specific system calls disabled(nerfed) within the runtime to prevent it from doing damage. The system calls that are removed from the python runtime prevent it from destroying IoT device flash disks, and since my vms don't mount or use flash disks it has little risk of doing much harm in this local network with vm's I plan to destroy after building the **intrusion** training dataset.
+
+.. note:: In the future, I highly recommend setting up **firewalls**, **ACLs** and isolated networking **subnets** to replicate actual production networking environments.
+
+Intrusion Dataset
+=================
+
+Here is how the **intrusion malware** dataset was created.
+
+Network Topology
+----------------
+
+Before starting any simulations, each vm has their own **/etc/hosts** synced to run the same shared network configuration:
+
+::
+
+    cat /etc/hosts | grep antinex
+    192.168.0.12    defend1 defend1.antinex
+    192.168.0.32    attack1 attack1.antinex
+    192.168.0.33    attack2 attack2.antinex
+    192.168.0.30    dev.antinex api.antinex.com webapp
+
+Roles
+-----
+
+.. list-table::
+   :header-rows: 1
+
+   * - Name
+     - IP Address
+     - Aliases
+     - Role
+   * - AntiNex Stack
+     - 192.168.0.30
+     - dev.antinex api.antinex.com webapp
+     - `Running Stack <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/master/compose.yml>`__
+   * - Defender 1
+     - 192.168.0.12
+     - defend1 defend1.antinex
+     - Defend
+   * - Attacker 1
+     - 192.168.0.32
+     - attack1 attack1.antinex
+     - Attack
+   * - Attacker 2
+     - 192.168.0.33
+     - attack2 attack2.antinex
+     - Attack
+
+Defend Role
+-----------
+
+Here is what the defend roles are doing during a simulation.
+
+#.  Set up the Honeypot
+
+    **Defender 1** is running the following `AntiNex capture tools <https://github.com/jay-johnson/network-pipeline#capture-network-traffic>`__ for recording the following incoming networking traffic:
+
+    - `capture_ssh.py <https://github.com/jay-johnson/network-pipeline/blob/master/network_pipeline/scripts/capture_ssh.py>`__
+    - `capture_telnet.py <https://github.com/jay-johnson/network-pipeline/blob/master/network_pipeline/scripts/capture_telnet.py>`__
+
+    .. note:: Please make sure to **configure the packet forwarding address** using the **FORWARD_BROKER_URL** environment variable before running the capture tools like: ``export FORWARD_BROKER_URL=redis://dev.antinex:6379/0``
+
+#.  Start Services
+
+    It will be hosting an ssh server listening on port **22**. It will also host a tcp server listening on the default telnet port: **23**.
+
+    .. note:: Make sure to generate new ssh keys and distribute them only on defenders.
+
+Attack Role
+-----------
+
+Running the `intrusion malware <https://github.com/jay-johnson/nerfball>`_ targeting the defend roles by alias and IP.
+
+.. note:: Make sure to generate new ssh keys and distribute them only on attackers.
+
+AntiNex Stack
+-------------
+
+Running the latest AntiNex stack of docker containers from the `REST API compose file <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/master/compose.yml>`__ which can be started using `these directions <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt#getting-started-with-docker>`__. With the stack running, you can connect the **capture tools** to publish traffic to the redis server listenining on port ``6379``.
+
+The stack is composed of:
+
+- `AntiNex REST API written in Django REST Framework <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L71-L87>`__
+- `AntiNex Celery Worker for the Django REST Framework <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L89-L104>`__
+- `AntiNex Celery Worker for Training and reusing Pre-trained Deep Neural Networks <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L54-L69>`__
+- `AntiNex Network Pipeline for Processing Captured Network Packets <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L106-L127>`__
+- `Redis for Caching and Publisher-Subscriber Messaging <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L29-L35>`__
+- `Jupyter <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L37-L52>`__
+- `Postgres <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L5-L15>`__
+- `pgAdmin <https://github.com/jay-johnson/train-ai-with-django-swagger-jwt/blob/0abe7e02122d8aa9dcd9bc9f23cfa9b889c886fe/compose.yml#L17-L27>`__
+
+Cleanup
+=======
+
+After using this test bed, it's easy enough to just delete all vms and vm disks to prevent re-use after the tests are completed.
+
+Legal / Disclaimers
+===================
+
+- This is an emerging technology, it will continue to change and there is no assumption the process here will work at all with future types of malware.
+
+- This guide is for educational purposes using an approach to observe how a piece of software, **that is claimed to do lots and lots of property damage**, works over the network. It was written to help share with anyone that wants to help try and learn how to defend against it and other bad actors better in the future.
+
+- I am not legally responsible for any damages using this guide causes with any type of malware. Just because this guide and approach works one time on my simple test bed, does not mean it will work on the same malware again or with different malware anywhere else in the future.
+
+- Assume any property you use during this guide needs to be thoroughly deleted and cleaned. None of the vms should be reused or started again after they are part of these malware simulations.
+
+- Assume you can lose any and all of the property used during this guide before starting. Whenever you are trying to observe known malware or untrusted code of any kind (including this code!) you have to accept those risks before continuing.
+
+- Use this guide at your own risk. I am not legally responsible for any damages this guide causes. Avoid using this approach and guide if you are not 100% comfortable trying to play with something that can destroy property, get you fired, jail or worse.
+
+- I do not claim this guide can be effective for building training data with any types of malware.
+
+- I do not claim this guide can be effective for all types of malware.
+