===========
Development
===========
Install for Development
=======================
First off, thanks for taking the time to contribute!
This small guideline may help takinf the first steps.
Happy hacking :)
Fork the Repository
-------------------
Clone pyftpsync to a local folder and checkout the branch you want to work on::
$ git clone git@github.com:mar10/pyftpsync.git
$ cd pyftpsync
$ git checkout my_branch
Work in a Virtual Environment
-----------------------------
Install Python
^^^^^^^^^^^^^^
`Python 3.8+ `_,
and `pip `_ on our system.
If you want to run tests on *all* supported platforms, install
Python 3.8 through 3.12.
Create and Activate the Virtual Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Linux / macOS
"""""""""""""
On Linux and macOS we recommend to use `pipenv `_
to make this easy::
$ cd /path/to/pyftpsync
$ pipenv shell
bash-3.2$
Windows
"""""""
Alternatively (especially on Windows), use `virtualenv `_
to create and activate the virtual environment.
For example using Python's builtin ``venv`` (instead of ``virtualenvwrapper``)
in a Windows PowerShell::
> cd /path/pyftpsync
> py -3.10 -m venv c:\env\pyftpsync_py310
> c:\env\pyftpsync_py36\Scripts\Activate.ps1
(pyftpsync_py310) $
Install Requirements
^^^^^^^^^^^^^^^^^^^^
Now that the new environment exists and is activated, we can setup the
requirements::
$ pip install -r requirements-dev.txt
and install pyftpsync to run from source code::
$ pip install -e .
.. $ python setup.py develop
The code should now run::
$ pyftpsync --version
$ 2.0.0
The test suite should run as well::
$ python setup.py test
$ pytest -v -rs
Build Sphinx documentation::
$ python setup.py sphinx
Run Tests
=========
The unit tests create fixtures in a special folder. By default, a temporary folder
is created on every test run, but it is recommended to define a location using the
``PYFTPSYNC_TEST_FOLDER`` environment variable, for example::
export PYFTPSYNC_TEST_FOLDER=/Users/USER/pyftpsync_test
Run all tests with coverage report. Results are written to /htmlcov/index.html::
$ pytest -v -rsx --cov=ftpsync --cov-report=html
Run selective tests::
$ pytest -v -rsx -k FtpBidirSyncTest
$ pytest -v -rsx -k "FtpBidirSyncTest and test_default"
$ pytest -v -rsx -m benchmark
Run tests on multiple Python versions using `tox `_
(need to install those Python versions first)::
$ tox
$ tox -e py36
In order to run realistic tests through an FTP server, we need a setup that publishes
a folder that is also accessible using file-system methods.
This can be achieved by configuring an FTP server to allow access to the `remote`
folder::
/
local/
folder1/
file1_1.txt
...
file1.txt
...
remote/ # <- FTP server should publish this folder as
...
The test suite checks if ``PYFTPSYNC_TEST_FTP_URL`` is defined and accessible.
Otherwise FTP tests will be skipped.
For example, environment variables may look like this, assuming the FTP server is rooted
at the user's home directory::
export PYFTPSYNC_TEST_FOLDER=/Users/USER/pyftpsync_test
export PYFTPSYNC_TEST_FTP_URL=ftp://USER:PASSWORD@localhost/pyftpsync_test/remote
This environment variable may be set to generate ``.pyftpsync-meta`` files in a
larger, but more readable format::
export PYFTPSYNC_VERBOSE_META=True
.pyftpsyncrc
------------
Instead of using environment variables, it is recommended to create a ``.pyftsyncrc``
file in the user's home directory::
[test]
folder = /Users/USER/pyftpsync_test
ftp_url = ftp://USER:PASSWORD@localhost/pyftpsync_test/remote
[debug]
verbose_meta = True
Settings from environment variables still take precedence.
Run Manual Tests
----------------
In order to run the command line script against a defined test scenario, we can use the
``tests.fixture_tools`` helper function to set up the default fixture::
$ python -m tests.fixture_tools
Created fixtures at /Users/USER/test_pyftpsync
$ ls -al /Users/USER/test_pyftpsync
total 0
drwxrwxrwx 4 martin staff 136 7 Okt 15:32 .
drwxr-xr-x 7 martin staff 238 20 Aug 20:26 ..
drwxr-xr-x 19 martin staff 646 7 Okt 15:32 local
drwxr-xr-x 18 martin staff 612 7 Okt 15:32 remote
The fixture set's up files with defined time stamps (2014-01-01) and already contains
meta data, so conflicts can be detected::
Local (UTC) Remote (UTC)
------------------------------------------------------------------------------
file1.txt 12:00 12:00 (unmodified)
file2.txt 13:00 12:00
file3.txt x 12:00
file4.txt 12:00 13:00
file5.txt 12:00 x
file6.txt 13:00 13:00:05 CONFLICT!
file7.txt 13:00:05 13:00 CONFLICT!
file8.txt x 13:00 CONFLICT!
file9.txt 13:00 x CONFLICT!
folder1/file1_1.txt 12.00 12:00 (unmodified)
folder2/file2_1.txt 13.00 12:00
folder3/file3_1.txt x 12:00 (folder deleted)
folder4/file4_1.txt x 13:00 (*) undetected CONFLICT!
folder5/file5_1.txt 12:00 13:00
folder6/file6_1.txt 12:00 x (folder deleted)
folder7/file7_1.txt 13:00 x (*) undetected CONFLICT!
new_file1.txt 13:00 -
new_file2.txt - 13:00
new_file3.txt 13:00 13:00 (same size)
new_file4.txt 13:00 13:00 CONFLICT! (different size)
new_file5.txt 13:00 13:00:05 CONFLICT!
new_file6.txt 13:00:05 13:00 CONFLICT!
NOTE: (*) currently conflicts are NOT detected, when a file is edited on one
target and the parent folder is removed on the peer target.
The folder will be removed on sync!
Now run pyftpsync with arbitrary options, passing local and remote folders as targets,
for example::
$ pyftpsync -v sync /Users/USER/test_pyftpsync/local /Users/USER/test_pyftpsync/remote
If an FTP server was configured, we can also run the script against it::
$ pyftpsync -v sync /Users/USER/test_pyftpsync/local ftp://localhost/Users/USER/test_pyftpsync/remote
Run ``python -m tests.fixture_tools`` again to reset the tests folders.
Run FTP Server
--------------
Run ``pylibdftp`` FTP Server Locally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In develpoment mode, pyftpsync installs `pyftpdlib `_
which can be used to run an FTP server for testing.
We allow anonymous access and use a custom port > 1024, so we don't need to sudo::
$ python -m pyftpdlib -p 8021 -w -d /Users/USER/test_pyftpsync/remote
or::
$ python -m tests.ftp_server
Also set the test options accordingly in ``.pyftpsyncrc``::
[test]
folder = /Users/USER/pyftpsync_test
ftp_url = ftp://anonymous:@localhost:8021
Run Built-in FTP Server on macOS Sierra
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Note:** This does **not** work anymore with macOS *High* Sierra.
On OSX (starting with Sierra) the built-in FTP server needs to be activated like so::
$ sudo -s launchctl load -w /System/Library/LaunchDaemons/ftp.plist
It can be stopped the same way::
$ sudo -s launchctl unload -w /System/Library/LaunchDaemons/ftp.plist
The FTP server exposes the whole file system, so the URL must start from root::
[test]
folder = /Users/USER/pyftpsync_test
ftp_url = ftp://USER:PASSWORD@localhost/Users/USER/pyftpsync_test/remote
.. warning::
Exposing the file system is dangerous! Make sure to stop the FTP server after testing.
Run FTP Server on Windows
^^^^^^^^^^^^^^^^^^^^^^^^^
On Windows the
`Filezilla Server `_
may be a good choice.
Code
====
.. note::
Follow the Style Guide, basically
`PEP 8 `_.
Failing tests or not follwing PEP 8 will break builds on
`CI/CD `_,
so run ``$ pytest``, ``$ flake8``, and ``$ tox`` frequently and before you commit!
Create a Pull Request
=====================
.. todo::
TODO