11 Commits (54cee443814c7a1cfb173619ce185ec54f6d94de)

Author	SHA1	Message	Date
Bryan Biedenkapp	47fff42173	add support for FNE tests via dvmtests harness;	3 months ago
Bryan Biedenkapp	399992fec9	correct possible edge case causing the Rx P25 stream ID to become stuck; build and implement a test harness against the host controller paths to validate logic where possible;	3 months ago
Bryan Biedenkapp	8fcd066c02	correct 3-bit error handling for Golay(24,12,8); add tests against network engine code; correct sendmmsg implementation against documentation for sendmmsg to better handle partially sent queues instead of discarding;	4 months ago
Bryan Biedenkapp	274b805517	Merge R05A04 (r05a04_dev) into master (#114) ... * rebuild r05a04_dev from nasty Git merge bullshit; implement handling of SNDCP on the FNE instead of dvmhost; add quick sanity Catch2 testcases; BUGFIX: NXDN SACCH was incorrectly handling the RAN and structure causing the structure value to become overwritten; correct badly set example IP range in FNE config; add AI generated documentation for the network statck, FNE REST and DVMHost REST; update version number for next dev version; * remove old documents; * BUGFIX: fix some unbounded array checking; * enhance InfluxDB implementation: fix bad handling on Windows; fix bad use of free() for address info freeaddrinfo(); fix partial data writes for bigger payloads; enhance getting responses back from InfluxDB (this makes it so that InfluxDB cannot fail silently easily); * EXPERIMENTAL: add an adaptive jitter buffer implementation to the FNE to better deal with peers on bad connections that may send packets out-of-sequence; * remove unnecessary garbage in AI generated doc; use references for autos when looping peers; normalize jitter buffer log messages; * cleanup doc; * properly move hastily erected debug YAML config to the peer ACL file instead; * update jitter buffer doc; * BUGFIX: correct naive implementation that did not consider the length of the packet being sent, which could result in a nullptr after casting the ip struct; BUGFIX: send actual bytes read from VTUN vs the default MTU size; * BUGFIX: correct similar ip struct casting problem; * implement a very preliminary DMR to VTUN IP dispatcher based on the P25 implementation (while I am sure this is probably going to be quite iffy, I am quite sure the VTUN TAP interface method we are using *is* the right direction; even though the P25 implementation is iffy, with iteration over time I suspect it is the correct approach, as such I have implemented a DMR equiviliant based on the P25 implementation); * use 3/4 rate instead of 1 rate; add some dev notes future me; * implement preliminary positive response to KMM HELLO (this still will fail with No-Service due to missing implementation for per-SU UKEK implementation from the crypto container); * add some extra logging for VTUN PDU; * better implement V(R) V(S) checking; * add prereq logging of last load time for various lookup tables; * add fne-stats; add reload peers and crypto; * update FNE REST API doc; * WIN32: fix ctime_r and add Win32 compat for ctime_s so we can compile on Win32 again; * Win32: FNE's P25PacketData clock() in its entirety should do nothing on Windows; * shut the compiler up about order of constructor initializers; * EXPERIMENTAL: add option for 1 second delayed network TDU when a subscriber dekeys; * slight TUI color palette change; * implement support to configure adaptive jitter buffer from peered; * initialize FinalCut palette earlier; * document that key IDs are hex not dec; * BUGFIX: lets not memset past the possible end of the key array....; * track active call counts (this might be slightly error prone, so we will at best call this an approximate number); * attempt to validate source IP/port during a RPTL, RPTK or RPTC event when RPTL, RPTK or RPTC is sent while not in the correct login state; * BUGFIX: dvmpatch did not properly release the network stream ID at the termination of a call, resulting in a hang up; * BUGFIX: for R05A04s 1s delayed TDU, make sure we have valid IDs before allowing a delayed TDU; * BUGFIX: add more stream ID lockout reset handling; * BUGFIX: add more stream ID lockout reset handling; * add some experiemental support to dvmpatch to support clear to enc one-way patching; * BUGFIX: only consider jitter buffer for RTP frames not carrying the end of call sequence (65535); * BUGFIX: always allow the network stack to process and pass RTP frames carrying the end of call sequence (65535); * BUGFIX: dvmbridge, ignore UDP frame timeouts when using RTP or USRP UDP streams; * add dropped call support to dvmpatch; refactor/rewrite dvmpatch encryption support; * hide debug messages; * BUGFIX: validate destination matches before handling TDU; * report ssrc for late packets; * prevent issue with 0 callDstId on dvmpatch; reject TDUs with no destination (these are invalid and can fuck off); * add separate packet dumping option to the network stack, to allow for less-verbose debugging messaging (debugigng where packet dump/trace is not required); * BUGFIX: fix condition in P25 call handler where if the TSBK was not decoded it could result in a crash; * add network debug trace around stream ID reset calls; * better handle framequeue debugging when not performing packet dumping; * implement proper packet reordering for P25 PDU data; BUGFIX: add boundary checking for P25 PDU disassembly; * BUGFIX: fix startup crash of dvmbridge when using analog audio mode; add transmit voice frame status for analog audio; correct core network issue determining length of analog audio frames; * bump copyright dates; * BUGFIX: for unauthorized, unconnected failures the NAK response was not sending the appropriate RTP sequence; * enhance REST API logging; add dev REST API commands to force send NAK packets; * add PDU block reordering support to dvmhost; * minor bugfix, correct incorrect log module for host sourced analog audio; * very experimental fix for #111, along with additional enhancements to repeat parrot traffic to all VCs of a trunk site to ensure parrot traffic repeats; * update bridge config and expressly document the need to properly configure timing parameters; * make the program status variable m_running a static s_running for dvmpatch; ensure PeerNetwork for dvmpatch uses m_packetDump instead of m_debug for packet dumping; * (we're still hunting the cause of bridge ceasing UDP socket operations after a period of time, it *seems* to occur on Win32 only though); make the program status variable m_running a static s_running for dvmbridge; ensure PeerNetwork for dvmbridge uses m_packetDump instead of m_debug for packet dumping; fix shutdown issue with local audio causing a crash for dvmbridge; refactor how RTP and uLaw frames work; remove and deprecate the no include length configuration parameter; refactor and centralize UDP audio writing functionality; add verbose log messages for if the s_running flag becomes non-true during operation; * fix concurrency lockup for DMR granting; * remove main thread sleep control this causes WinSock on Win32 to lose its mind when polled too fast; * move sample level dumping option to CLI; * always reset the call streams at the end of a call; * increase packet processing latency warning to 250ms (1/4 second); * begin adding support for P25P2 network transiting; * add some verbose debug trace for checkNetTrafficCollision(); * begin defining P25 Phase 2 primitives; * P2 abbreviated/explicit partition MCOs are essentially just TSBK opcodes; * add MFID specific partition type; * refactor MAC MCO defines; * correct comment; * make slot numerical; * whoops meant == not =; * implement RTP audio pacing by timestamp sent by source; * for maintainability split modes into separate source CPP files; BUGFIX: fix buffer overflow when copying PCM data; allow uLaw encoded RTP frames to also carry source and dest metadata; * preamble tones are not supported for UDP audio; * BUGFIX: dvmpatch do not send LDU2s until network is in non-idle state; * convert direct MMDVM TDU to a timed TDU; correct some handling of LDU1 srcId and dstId; * remove dvmmon, R05A02 will be the last version to support dvmmon, R05A04+ will not support it in favor of using sysview from the FNE for monitoring; * add more P25 Phase 2 primitives for S-OEMI sync and Hamming (8,4,4) for P2 DUID handling; * more P25 Phase 2 constants; * extreme preliminary work to handle P25 Phase 2 MAC messages (nothing uses this yet this is for future proofing); * continued work on preliminary instrumentation for Phase 2 MAC handling; * fix missing parens; * split the decodeVCH_MACPDU into two functions, one for IEMI the other for OEMI; * bugfix and correct encode/decode for RS 52,30,23, 46,26,21, 45,26,20, 44,16,29; implement catch2 cases for P25 Phase 2 RS codes; correct bad EC check for RS 24,12,13, 24,16,9 and 36,20,17 that could cause false positive failures in edge case conditions; * implement and add more EDAC verification test; * add DMR EMB and QR 16,7,6 tests; correct issue where QR 16,7,6 decode() was correcting and returning the wrong number of bits; correct issue where DMR EMB would not actually use the corrected QR 16,7,6 codeword; * finally what I was trying to get to -- add testcases for the new P2 MAC PDU logic; correct implementation problems with the P2 MAC PDU handling for OEMI and IEMI (the implementation still requires handling scrambling); * add missing Hamming FEC test cases; * finish out the base crypto test suite and include the DES algo; * (this first part is subject to Git revert if it becomes problematic, but because this is a dev branch I am gonna roll with it) revert FrameQueue timestamp list/map changes back to pre-R04J32, the original implementation was far cleaner and faster with O(1) speed for timestamp lookup vs the O(n) lookup, additionally, the newer R04J32+ implementation introduces a pointer that can be dereferenced incorrectly causing a edge case crash; fix the implementation for handling the mutex for timestamp map locking, the implementation used a incorrect instance mutex which could in high-traffic conditions cause a race condition and ultimately a crash; * for condition where a downstream peer is trying to connect to us, only validate the IP instead of IP/port; * EXPERIMENTAL (and untested): allow source ID overriding for parrot playback, this feature will rewrite the source ID to a static ID configured in the FNE configuration, for P25, NXDN and Analog this rewrite is striaghtfoward, for DMR this will likely work oddly due to the way DMR has source ID data embedded in the transited data frames; * validate the RPC data length includes the message; * BUGFIX: handle edge case where a opcode $27 is not an ISP_EMERG_ALARM_REQ but rather its a network sourced OSP_DENY_RSP; BUGFIX: handle incorrect decoding of OSP_DENY_RSP by SysView; BUGFIX: for DVRS operation properly set MFG_MOT for DENY and QUEUE outbounds; * do not use [] access for grantTimers and unitRegTimers, use find() lookups these are safer; * issue a notify CC channel release on a ChanRelease TDULC; * add explicit TDU TG release option, this option allows a CC to process incoming TDUs to determine if a channel grant should be released; add more explicit TDU process logging to the FNE, ensure TDUs being sent outside a call are being logged properly; during a TG unpermit on a non-authoritative VC, transmit a burst of TDUs on the outbound VC RF interface; add extra verbosity to the active TG logging; * add option displayModemDebugMessages to optionally disable or enable debug log messages coming *from* the modem; properly check if we're operating in DFSI or not and if we are do not attempt to set the FIFO buffer lengths or clear buffers related to DMR or NXDN; correct deletion of array types in the DFSICallData structure; correct bad length of the VHDR1 during DFSICallData initialization, the length defaulted to TIA mode which would cause a crash when trying to delete and deallocate VHDR1 when used in V.24 mode; * correct TSBK, TDULC and PDU transmission using V.24 DFSI, timing must be applied like IMBE; lengthen the high-level CC generation timer by 10ms, this has no impact on regular air and hotspot modems, but better times CC frames for V.24 DFSI modems; refactor naming to be clearer for the V.24 jitter buffer transmit types; because V.24 uses OSP_TSBK_RAW and LC_TDULC_RAW correct several off by 2 shift bugs; remove unnecessary V.24 start of stream and end of stream calls; * allow TDUs to be sent regardless of whether a V.24 call was started or not; * do work on V.24 PDU support (inbound should be working pretty much perfect now, outbound -- is still broken); * V.24 PDU work; * allow the user to adjust the frame timeout length when using frame timing at the bridge; ensure audio frames aren't greater then x2 the size of an expected audio frame; * after consideration entirely refactor how UDP frame timing is generated internally when we're using raw PCM; fix a variety of bugs related to locking and handling of the deque for UDP audio frames; better insert silence at the start and end of calls; * disable this debug trace, its too noisy; * do not consider active call or call counts for parrot; * add call collisions to the stats counters; add REST APIs to reset and zero call counters; * log non-call DMR terminators like P25; * correct endpoints for counter reset APIs; * reset active call count on a maintainence loop; * increase V.24 outbound rate for TSDUs; * properly set the V.24 Tx buffer size to the FIFO length like done for air and hotspot modems; pass whether or not the frame being written to the modem is an immediate frame; modify V.24 modem to properly maintain two independant buffers, one for background/normal priority frames, and one for immediate priority frames (this helps with CC mode delays, there are more buffering issues, likely at the modem to mitgate but thats a future project); * update gitignore to include some python stuff; add very very preliminary Python tool that helps generate dvmhost configurations; * add option to disable deny responses in conventional for radios that are braindead and interpret an OSP_DENY as a ISP_EMERG because of opcode reuse (thanks TIA-102); * delete array instead of delete; * add protections to prevent the network mutex from locking too long; add proper network watchdog to prevent unterminated calls from hanging bridge; * ICC and GRANT shouldnt NAK, as this will cause a straight disconnect/reconnect, ignore them instead; * fix issue where in-call control might start looping requests; * lets make sure RID0 is never used, treat as WUID_FNE instead; * only send ICC upstream when the ICC request originates on the local FNE; * Revise cross-compilation instructions in README Updated cross-compilation instructions for ARM 32-bit and ARM 64-bit, including examples and required packages. * relabel some source files in the FNE, FNENetwork -> TrafficNetwork, DiagNetwork -> MetadataNetwork; remove support for disabling the alternate port, this feature is mandatory now; * identify a software SDR; * add support to dvmcfggen for logging configuration; add support to dvmcfggen for user supplied answers files to automate skipping certain wizard questions; * typo; * HIGHLY_UNECESSARY_DISCLAIMER_FOR_THE_MENTAL; * the typo of that define was driving my OCD nuts; * include new statement in README.md; * enforce a users usage understanding by adding a startup configuration parameter that must be set; * these flags are root level; * make not be stupid error more clear; * copy legalese from config.yml to README.md; * policy updates * Update usage_guidelines.md * create issue & pr templates * remove dvmmon from bug & pr template * revert how timestamping is generated; * BUGFIX: fix incorrectly set length for RTP packets, we were incorrectly adding 4U bytes which does not apply to RTP frames; * enrage the entire community by cleaning up the bridge configuration to be more sane -- a lot of these parameters had nothing to do with FNE network configuration; * code cleanup (no functional change); * fix missing marker bit on RTP seq 0 to make jsb stop complaining; fix missing CTS controller teardown; * properly set the FIRST seq (when seq is reset to 0) as a marker; because JSB is *REALLY* hell bent on this, implement continuous RTP seq to make him happy; * wtf; * cleanup typo and bad spacing due to bad VSCode configuration; correct memory leak when handling buffer drop due to new stream packet wraparound; * correct issue where RTP sequences were not stepped appropriately; * ignore m_audioDetect flag for UDP audio outbound to the FNE; attempt to correct becoming stuck on incoming RTP timestamps; * whoops remove debug trace line; * remove incorrect shift; * ensure network watchdog resets all states; ignore TDU/TDULC when local audio or UDP traffic is running; * reset callStartTime for TGs who receive Non-Call TDU/Terminators; * remove duration log from Non-Call TDU; * disable the expected TS code for RTP frames; * add mutex locking when making a copy of the RadioId table to prevent thread-based race conditions; --------- Co-authored-by: Dev_Ranger <30966416+DevRanger@users.noreply.github.com>	9 months ago
Bryan Biedenkapp	abe45a1d53	fix up file headers;	2 years ago
Bryan Biedenkapp	5a5104ba2f	treewide: perform general refactor of file headers, I've modified the file headers to follow rules from: https://www.kernel.org/doc/html/next/process/license-rules.html, this trims/compresses the file header section a bit reducing the number of lines at the top of every file quite a bit, going forward all new files should follow this new format (if its good enough for the Linux kernel, its good enough for me);	3 years ago
Bryan Biedenkapp	140b4ecd3c	further cleanup/rework of CMake scripts;	3 years ago
Bryan Biedenkapp	1379482fcc	reorganize entire codebase structure; move common classes, files into a compile time static library; reorganize CMake files to make use of common static library to reduce compile time;	3 years ago
Bryan Biedenkapp	243ad6ce0c	increase NXDN voice call verbosity; correct incorrectly calculated data offset for NXDN voice data;	3 years ago
Bryan Biedenkapp	1eb24608a0	add tiny AES library (for future LLA use);	3 years ago
Bryan Biedenkapp	2b6b0c574d	merge 3.5-dev into master, this marks 3.5 alpha/beta release; (#31) ... * implement inital code to support embedded FNE; * continued implementation of embedded FNE; * implement initial core functionality for handling FNE side of DMRD, P25D and NXDD; * update README.md; * refactor and rename files; clarify that the embedded FNE is not a "routing" FNE, but rather, a "conference bridge" FNE; * more cleanups making the conference bridge FNE implementation clearer; * implement proper code for handling peer inclusion/exclusion; implement proper code for validating a call stream; * add missing comments; * make the GCC compiler happy when compiling on 32-bit ARM; make old RPI_ARM compiler happy; * process parrot flag for talkgroup configuration; * deprecate and remove old tg_acl.dat file infavor of the YAML-based talkgroup rules file for the entire host; deprecate and remove custom Mutex class in favor of std::mutex; expand helper routines for the TalkgroupRulesLookup class (backwards compatibility essentially with TalkgroupIdLookup); fix issue in network core that could cause deactivated TGs to try using talkgroup lookups when talkgroup lookups may be unavailable; * support code for future use; * fixup CMakeLists to fix build warnings for ARM 32-bit platform; * update detection for sendmsg and sendmmsg (handles some platforms that don't define sendmmsg); implement container classes for RTP framing; * update CMakeLists to handle new files appropriately; * initial implementation of FrameQueue to generate FNE RTP frames; * fix incorrect return; * implement a typedef for unique_ptr uint8_t arrays; fix issue where DMR, P25 and NXDN control classes were accepting BaseNetwork instead of Network; refactor how FrameQueue handles returning messages read from the network; refactor BaseNetwork, Network and FNENetwork to use FrameQueue instead of raw network writes; * fix issue with handling network state; fix issue handling global network enable flag; fix buffer pointer cleanup; * add option to CMake to selectively utilize the legacy (non-RTP) network protocol (embedded FNE does not support this and will be disabled for USE_LEGACY_NETWORK; cleanup some error messages when trying to initialize and read the configuration file; fix some minor issues in parsing the new talkgroup rules YAML file; complete conversion of BaseNetwork from raw socket to using the new FrameQueue class; refactor the multi-buffer UDPSocket write function to properly send multiple messages in batch down to the kernel driver (on Linux using sendmmsg); add flag to UDPSocket to indicate whether the socket is "open" or not; correct some RTP formatting issues with the extension header (the payload length is in 32-bit units); * remove remaining native WIN32/WIN64 type support code (it was unmaintained anyway); correct issues with network socket reading; cleanup some error messages;q * finally resolve lingering issue with frame queue having random socket issues; * refactor how BufferVector and sendmmsg UDPSocket write operates; refactor FrameQueue to properly queue messages; * properly queue packets to send in bursts to peers; add missing ACKs; add missing MSTPONG; * fix up formatting; remove unnecessary debug statements; * minor cleanups; * remove unused function; * add fields for function and sub-function; * describe network functions and subfunctions; * reorganize some code; add checking for packet sequence; * roll next sequence properly; * utilize typedef for std::unique_ptr<uint8_t[]>; * fix usage of __UNIQUE_BUFFER to __UNIQUE_UINT8_ARRAY; * clean up macro definitions; * implement sequence counting for the FNE conference bridge; * remove USE_LEGACY_NETWORK support; * use macro expansions for this instead of raw buffer lists; * implement actual transmission of new network func/sub-func behavior; * deprecate use of packet tags for determining operation, instead use RTP FNE header function and subfunction bytes; * fix issue with send talkgroups not setting flag to send talkgroups to peers; fix issue when compiling list of activated or deactivated TGs to send to peers; correct strangeness with the RTP sequence counting; fix talkgroup rule lookup on peer when activating or deactivating TGs; * update copyright dates; * complain if the RTP and FNE headers don't agree on stream ID; * for when you forget to update a comment so things actually make sense when you look at it later...; * remove extraneous check, this is really not necessary since FrameQueue does the size validation; * clarify the description of the "embedded FNE"; * merge changes from https://github.com/CVSoft/dvmhost to place spec specific limits on channel spacing; * refactor how network traffic is handled, we don't handle protocol specific stuff within BaseNetwork anymore and offload that to the specific protocol controller instead; implement support to receive peer traffic and repeat (still need to do master to peer); * convert fatal error about Rx frequency being below base frequency to a warning instead; * fix incorrect redefine of slotNo; * correct ret flag for readDMR, readP25 and readNXDN not being set to true by default (stupid me); correct FNE-mode payload offsets for diagnostic packets; * implement option to selectively enable/disable FNE mode verbose logging information; enhance FNE display of network diagnostic logs; refactor dmr::Control::processNetwork(); refactor RTP packet sequence counting; * fix incorrect variable used for out-of-sequence log message; * fix issues with login RTP stream; specially handle certain control opcodes with a different payload type; continued enhancements to sequence counting; * whoops readd accidentally removed block; * backout previous lastPeerId code; properly ensure RTP packet sequence is repeated for traffic calls; * transmit DVM LC_CALL_TERM TSDU at the end of call or if a call drops (hopefully); * don't reset the packet sequence for a RPTPING; * rename some class variables for clarity and prep for some future changes; * add support for priority/immediate Tx queues, these queues are intended to be used sparingly (and mostly for control data) to transmit priority messages [this is an experimental change and may break the build!]; * fix issue where connecting to a non-existent REST API endpoint would crash with an unhandled ASIO exception; * increase timeout delay for the modem by 4 seconds (the original 4 second delay would be too short for some longer operations); * use the immediate queue to prioritize control data in some situations [this is experimental and may break things!]; * reformat headers; * implement toString() functions on TSBKs, CSBKs and RCCHs for future use to replace fixed strings; cleanup DMRDefines slightly; * fix up and simplify RTP timestamp generation; * reduce reliance on fixed strings, instead rely on toString() functions from TSBK, CSBK, and RCCH to ensure string consistency for logging purpsoes; * refactor how and when releaseGrant is called in some situations; ensure channel grant messages traverse the network; * fix reset of error count between network super-frames; * add development helper file for VS Code to configure debug launch parameters; * update CMakeLists to be more condusive to debugging; * cleanup AUTHORS.md; * implement parrot for the conference bridge FNE; implement appropriate call detection logic for conference bridge FNE and attempt to prevent call collisons for the same destination; * add support for altering softpot levels from calibration mode; * bump copyright information; * reorganize CMakeLists; * remove finalcut from the libraries list (this isn't supposed to be included yet!); * implement initial rework of setup/calibration mode into a TUI; add ENABLE_TUI_SUPPORT CMake option to disable *all* project TUI support; add ENABLE_TUI_SETUP CMake option to disable setup TUI support for dvmhost; * properly update isConnected flag; * fix issue with incorrectly placed control; * part 1 of a fix for #30, the change here would be to allow the VCs to know about the CC to be able to report back to it to release or other wise update grants; * part 2 of the fix for #30, this implements the actual logic for a VC to notify the CC of a grant status change (touch, release) [NOTE: this could have undesirable consequences and this commit is experimental!]; * update README.md; * fix terrible copy and paste job resulting in a poor logic check (SMH); print to log the control channel address and port at startup; use the appropriate REST API for touch (whoops); * remove transmit overlay on shell; add F12 hotkey to trigger transmit; implement support to use F2 and F12 from various configuration/calibration modals; correct some mode strangeness when saving; * implement support to adjust FIFO buffer sizes from the host; * add engineering menu to adjust FIFO buffer sizes from setup; * ensure set button state is set properly for these windows; * fix stupid --tags non-sense in the git hash reporting; * ensure certain error conditions for setup mode are displayed when exiting at initial startup; * more fixes to attempt to satisfy #30 (why is this being so difficult!); correct situation where RESTClient may throw an assert instead of handling an error condition nicely; * for #30 allow release tg grant and touch tg grant to work regardless of dedicated CC setting; * file cleanups; * C++11/14 cleanups, class overrides and pure-virtual implementors /worked/ but were not really correct; * stringify modem command and reason messages (so log errors are easier on the eyes); * late update during a branch merge, to fix some latent issues that were missed;	3 years ago