7 Commits (d68fffad971bc92abbbeb4e656dd03f8d72aec92)

Author	SHA1	Message	Date
Bryan Biedenkapp	274a8f23fc	Merge R05A02 (r04k32_dev branch) into Master (#110) ... Below is a summary from the various commits, this release updates the major version number. Backward compatibility, with previous versions should not be a problem. But YMMV and its important to note that FNE backward compatibility, specifically might be problematic (FNE's should be at the same release level). This release changes significant under the hood implementations, including the lowest level of our network stack. * update version number for next dev version; * implement passing complex types as refs in lambda functions; * move KMM NoService response into its own helper function; * remove data call collisions (this code was really iffy); refactor log messages; refactor default handling of packets the FNE doesn't process; insert null bits before PDU; * set RSI data properly for outgoing KMM frame; * implement TEK encryption with an AES-256 KEK; * add support for unwrapping (decrypting) a KEK encrypted TEK; * reorganize code and organization for handling P25 OTAR KMMs for better separation and robustness (this is still a WIP and does not function!); * force hard disable KMF services if OpenSSL isn't compiled in; add instance of P25 crypto to P25OTARService; * add support to expose KMF services via DLI UDP; add plumbing to support encrypted KMM frames; * add some debug dumping; * BUGFIX: null reference when trying to perform old style lookup of timestamp when debugging is enabled; * standardize KMM logging; * begin adding support for V.24 PDU data; * fix C++ namespace shenanigans; * fix documentation error; * add support for outgoing V.24 PDU data; * minor alterations to bridge UDP audio logic; set better default for bridge udpJitter buffer; * re-engineer entirely how source untimed raw PCM frames over UDP are handled and timed; refactor how udp end of call is handled; * add support to resize the recv and send buffers on the raw UDP socket; adjust the recv buffer on bridge to use 131K system buffer for the socket (this allows us to hold ~394 frames worth of *raw* PCM + metadata in the socket's internal buffer in the system kernel; * set socket buffer sizes to larger values then the default; * annotate Linux limiting the maximum send and recv socket buffer sizes * update dvmhost submodule; * use a 2M buffer for bridge UDP audio; * display the socket buffer resize as warnings and not errors; * reduce UDP recv/send buffer size to a lower reasonable value of 512K; * update README.md; * add more class copy safety; * correct missing parens; * correct loop indexing; remove unused variable; * deprecate unused DIU flag (this isn't what this byte meant in the first place); * add call start marker to main application log; * remove SIP classes (this will be done by a different team, and done differently); begin refactoring Log and ActivityLog implementations into C++-type functions to do away with the C-style va_args functions for log message handling; correct -Wstringop-trunction warnings from BaseNetwork.cpp (we hide the warning for these because we are intentionally copying these strings without the nul terminator); * whoops accidentally blew away activity log transmission to the FNE, fix that...; * add better concurrency protection to AffiliationLookup; fix issues when getting the granted source ID from a destination ID by properly ensuring the mapping table has an entry first; add srcId to the release grant callback (callback should never call AffiliationLookup gets as it can deadlock, so adding this parameter gives the source ID to the callback so that it doesn't have to do the lookup); * remove __spinlock() from touchGrant() and isGranted(); * correct bad ordering of log message for call source switching; * remove blockTrafficTo; implement promiscuous hub mode for FNE (this mode allows the FNE to pass any and all traffic transparently); * fix typo; * for the purposes of my OCD fix incorrect one-liner Doxygen documentation (its //!< and not just //!); * better document peer ID and rid ACL list files; * add backward for stacktrace support on crash; * fix FNE compilation when SSL is not available; make Win32 builds work again; * remove double error message; * generate stacktrace file if main logger file is not initialized or unavailable; * reduce the use of unordered_map::at(); * begin relabeling peer-link to peer replication/peer replica; * implement identity with qualifier, this makes logs (and only logs) easier to trace by uniquely identifying certain peer types, a peer qualifier is simply a symbol appended to the beginning of a resolved peer identity in the FNE logs (@ = SysView, + = External/Linked FNE, % = Replica FNE); silence the Call Source Switched log messages, we will only actually print these if the call source ID changes; fix missed Peer-Link log branding/naming; * lock m_status and m_statusPVCall before trying to update data elements; * don't drop out-of-order packets (this was a bad idea, instead log an issue, the real fix for this will be some sort of RTP jitter buffer); if the current packet sequence reaches the maximum allowed, roll over to 0; * update order of operations for peer ident; * normalize log messages; * differentiate a call end collision from a call collision; * differentiate a call grant collision from a call collision; * fix missing code commenting; * initial implementation of naive round-robin HA mechanism. this mechanism allows the FNE master to communicate to a peer connected to it, to announce alternate IPs for the peer to connect to if the primary configured FNE master becomes inaccessible. the HA mechanism requires peer replication to create a loose cluster of FNEs, each FNE in the cluster is configured with the external WAN IP and port for connection to the FNE, and these IPs are then disseminated to all FNE replicas (and downstream connected peers) in the cluster automatically; * hide debug messages unless debugging is enabled; * remove old call in progress global; refactor call collisions, allow the call collision timeout be user definable (with 0 disabling call collisions, user beware); * simplify sendmmsg implementation; * BUGFIX: correct scenario where traffic from an upstream master to a downstream peer FNE would lose the RTP sequence numbering; * refactor how RTP multiplexing by stream is handled; * fix some odd behavior with very fast calls locking up grants on on DVRS channels; * refactor RTP sequence count handling; * whoops this stream ID check was intended for non-promiscuous operation only; * fix memory leak with PacketBuffer::decode(), dont use a direct heap allocated buffer, instead use a unique_ptr buffer; * simplify implementation; * rename backtrace namespace to log_stacktrace, when using simple mode the namespace conflicts with a global function backtrace(); add package element for libdw-dev:arm64 for cross compile instructions; * whoops forgot to update BridgeMain.cpp for log_stacktrace; * lock the peer list when writing traffic data to prevent peer removal during traffic operations; * utilize a shared_timed_mutex for peer list locking on the FNE, this better keeps the peer list locked during traffic operations using lock counting vs the original spinlock mechanism; * better handling locking peer connections during critical state changes; * whoops this lock should only take place for a connected peer; * reflect higher requirements for FNE; fix issue on host where sometimes a stuck network call would cause network traffic to stop incorrectly hanging on the previous TG; * change around some naming; * typo; * [EXPERIMENTAL] implement rudimentary spanning tree mechanism to prevent peer looping; refactor how FNENetwork handled peer disconnect cleanups and consolidate into a singular routine; * [EXPERIMENTAL] enhance detection for case where FNE A and FNE B are cross-peered to each other (dont do this); * add REST API endpoint to fetch the master tree; * make sure master peer ID 0 doesnt ever happen; * log condition where masterPeerId is 0 * SysView peers announce themselves as an external FNE, but we do not consider them FNE peer links; * SysView masquarades its masterPeerId as itself; * instead of killing a peer connection instantly on a duplicate conn drop, increase retry time to 30 minutes and allow up to 3 duplicate conn failures before killing; * better handle duplicate connection disconnect NAKs; add REST API on FNE to force reset a upstream peer connection; * fix issue with purely ACL virtual FNEs not being able to replicate configuration further down the master tree; * deprecate the "external FNE" terming for upstream FNE connections and instead call them "neighbor FNE"; * more code cleanup, simplify naming; * cleanup code; refactor log messages from the FNE to better categorize them; correct issue where a peer reconnecting may trip tree duplicate conn checking; * enhance STP peer reconnect logic to allow peers announcing the same peerId and masterId to reconnect between spanning tree updates; * update log colorizer to match new logging categories in the FNE; * update log colorizer to match new logging categories in the FNE (round 2); * update log colorizer to match new logging categories in the FNE (round 3); * update log colorizer to match new logging categories in the FNE (round 4); * bump major version numbering in a preliminary fashion, at least until group talks about it are done (so this could be permenant); * fix messaging for fast peer reconnect (it was misleading as a RPTC NAK which it isnt); * allow reparenting of a STP node if it moves from one tree node to another; * implement tree cleanups if the downstream announcement removes child leaves or reports no children at all; * relabel MasterTree to SpanningTree proper; * whoops errant i++; * fix up some concurrency problems when dealing with parrot transmissions, due to migration of parrot playback into its own thread; * make sure to share lock peers while processing in maintainence loop; * add mutex locking around spanning tree updates; * simplify log levels, deprecate LogMessage log level in favor of just using LogInfoEx (message and info logs are basically the same thing); * better classify log messages; * simplify FNE configuration for peers, make identity a global applying to all peer connections, remove bogus frequency data; * better report global identity; * allow overriding the global identity for upstream connections; implement use of identity for spanning tree; * update config examples to reflect new log levels; * add back peer "name" field, this is strictly informational to make the config file easier on the brain; * Win32: bump version number for file resource metadata; * add documentation for packet payloads for: writeLogin, writeAuthorisation, writeConfig and writePing; * add some more packet payload documentation; * continue packet payload documenting; * minor comment alteration; * correct DMR data handling; refactor FNE DMR data calls to be structured more akin to P25 data calls; * use WASAPI by default on Windows; * better handle out of order blocks for PDUs; reduce packet retry to 2; correct handling ack response packets; * refactor how buffered UDP datagram queuing is performed; * relabel static class variables to use s_ and globals to use g_ * enhance colorize-host.sh to terminate color properly; * enhance colorize-host.sh to terminate color properly; * RTS PTT will only assert when audio is present. Add holdoff timer before removing RTS. (#103) * add option to enable/disable upstream call start/end event logging; * more work towards a working DLD-type OTAR service for P25; * fix typos; * Add carrier operated relay support to dvmbridge. (#104) * refactor KMMFrame to properly support message number and add framework to support MAC; refactor all derived KMM classes to properly determine KMM frame length and body offsets; change collision in class naming for KeyItem in the CryptoContainer and KeyItem in the p25::kmm namespace; cleanup magic numbers; add initial code for transmitting a rekey command; add KEK crypto wrapping testcase; * begin work on generating KMM CMAC for message authentication; * document sections of doc where a test originates test data; * begin work on generating CBC-MAC for message authentication; * minor cleanups; * test should be using generated key; * CBC-MAC now works properly; CMAC MAC generation works (just gotta fix CMAC MAC key generation); * complete CBC-MAC implemenation, at least for the provided samples from TIA-102.AACA-C this is now passing for the rekey example provided in section 14.3.4; anything compiling against libcommon.a while OpenSSL is enabled will require OpenSSL -lssl; fix delete vs free in various tests; * typo * implement OFB data encryption per TIA-102.AAAD-B; test proper final encrypted output for CBC-MAC check; implement use of cryptAES_PDU() in P25OTARService; * add experimental DES crypto; * move Git hash global defines out of Defines.h and into GitHash.h; * fix copyrights; * cleanup file headers containing lingering old comments; update fw/modem submodule; update fw/hotspot submodule; * remove debug trace not needed anymore; * add support for PDU auxiliary ES headers; * complete refactor of how P25 PDUs are assembled from user data and disassembled to user data; various corrections for data path nullptr reference issues; implement several tests for testing the P25 PDU assembler; * remove left over debug code; fix AMBT CRC-32 calculation error, AMBTs calculate the CRC-32 for the PDU themselves the Assembler does not need to do it; correct default value of p25TxNAC; * add some better error handling for NetRPC; * validate LC_CALL_TERM peer ID before allowing them to repeat, this fixes an issue where an errant peer on the network could spam LC_CALL_TERM to cause trunked nodes to terminate/kill a call in progress; * fix issue with dvmpatch not properly evaluating the destination TEK; * add support to patch to *disable* enc processing and allow frames to pass transparently; implement DES for bridge and patch; * reorganize application files, I've wanted to do this for a while JSON isn't really a network service, and REST API while it is a network service, I want to be in a separate namespace; * add multi-block PDU for V.24; * add some guard rails on PDU reception so we dont overflow ourselves; * it seems as if PDU_(UN)CONF_END is variable length, it will always be at least 1 block containing the last PDU block, and PDU_(UN)CONF_BLOCK_X is always 4 blocks; * cleanup dataBlocks pointer array; * preliminary code for sending PDU frames >3 blocks over V.24; * use unsigned numbers for these loops the values should never be negative anyway; * properly encode sequence opcode; * whoops missing parens; * initialize these unordered_maps to allow the max connection cap count number of entries (before it was zero and could result in many reallocations of these maps); * well that was a little naive of me, I didnt multiply the count by the size of the stored elements...; * reverse course lets do this more intelligently and implement the passthru for the reserve() function on the concurrency classes; * add STP check to see if a downstream leaf is blown itself away on the tree; * implement DMR data PDU assembler based on the P25 PDU assembler; * expand on the files related to an FNE instance for clarity; * add preliminary support for enabling peers with call priority, this will give those peers the ability to override any current call in progress; * implement a more solidified peer call priority mechanism using ICC; * bump WebSocketPP version to conform to new CMake minimum version; (#108) * fix typo; * add some documentation around newer protocol additions; * whoops typo; * start adding foundational work for future P25 Phase 2; * decomplicate foundational changes to RS coders for Phase 2; * added new peer config options to FNE REST api * add some locks around pkt maps in DiagNetwork; * better document the master peerId importance; * ensure buffers are set to nullptr after deleting to prevent double frees; * skip trying to transmit any buffer that is null; * add thread safties to SpanningTree; * correct bad memory allocation; * revert previous change to add locking to SpanningTree (this is handled externally in FNENetwork and DiagNetwork); fix issue in FNENetwork which could cause a STP deadlock; fix issue incorrectly labeling a peer as allowing call priority in the log when infact the peer was not configured that way; * make sure during resetPeer() we lock the connection; * readd SpanningTree internal locking mechanisms; * make sure pointers are set to null after delete; * ensure when a FNE loses all its downstream leaves, that it will properly notify upstream FNEs; fix issue where dangling tree nodes were being incorrectly left in the flat peer node list for the spanning tree; * fix defaultNetIdleTalkgroup being treated as a hex value instead of dec; enhance P25 defaultNetIdleTalkgroup slightly to better pass traffic after RF traffic; enhance P25 defaultNetIdleTalkgroup to pass traffic if there are affiliations to the group; * how about we dont blatently leave debug messages enabled...; * BUGFIX: hopefully correct crash condition when trying to erase child nodes; * followup for last commit, simplify implementation; * this will be unpopular, but I am deprecating support for cross-compiling for armhf/legacy RPi, maintaining this is already causing problems with OpenSSL and will ultimately handcuff us in the future if we upgrade C++ versions because the legacy toolchain uses GCC 4.9; * remove deprecated patch; * update README.md; * correct some cross-compile shennigans for OpenSSL; * correct some cross-compile shennigans for aarch64; * BUGFIX: fix issue where the host would incorrectly reset the voice stream ID during a call; * match code change from previous bugfix to DMR and NXDN; * dont use the deprecated OpenSSL1.1 functions, use portable AESCrypto functions instead for low-level AES crypto; * disable STP reparenting when deserializing children of a tree; * BUGFIX: fix tged and peered in the case where an empty file or file with no entries is provided for editing; * BUGFIX: add some validation checks around deserialization reparenting; * implement P25P2 reed-solomon codes for future use; * BUGFIX: correct bad check for parrot frames end of call; --------- Co-authored-by: Lorenzo L. Romero <lorenzolrom@gmail.com> Co-authored-by: Natalie <jelimoore@gmail.com> Co-authored-by: W3AXL <29879554+W3AXL@users.noreply.github.com>	2 months ago
Bryan Biedenkapp	d6cf8b1155	remove debug launch option for dvmmon from the launch.json contrib;	1 year ago
Bryan Biedenkapp	31ca17449d	*BEWARE* this commit alters the RPI_ARM cross-compiler and may break things *BEWARE*; implement support for HTTPS SSL REST API configuration; add contrib binary overlay for the RPI_ARM compiler (to add libssl-dev dependences);	2 years ago
Bryan Biedenkapp	f0be2e0079	fix incorrect variable set in P25 SiteData; add VIM YouCompleteMe project configuration;	2 years ago
Bryan Biedenkapp	596fad3e30	update vscode launch.json file;	2 years ago
Bryan Biedenkapp	6c230995df	add c_cpp_properties.json for better VSCode support; update debug types for VSCode launch.json;	3 years ago
Bryan Biedenkapp	2b6b0c574d	merge 3.5-dev into master, this marks 3.5 alpha/beta release; (#31) ... * implement inital code to support embedded FNE; * continued implementation of embedded FNE; * implement initial core functionality for handling FNE side of DMRD, P25D and NXDD; * update README.md; * refactor and rename files; clarify that the embedded FNE is not a "routing" FNE, but rather, a "conference bridge" FNE; * more cleanups making the conference bridge FNE implementation clearer; * implement proper code for handling peer inclusion/exclusion; implement proper code for validating a call stream; * add missing comments; * make the GCC compiler happy when compiling on 32-bit ARM; make old RPI_ARM compiler happy; * process parrot flag for talkgroup configuration; * deprecate and remove old tg_acl.dat file infavor of the YAML-based talkgroup rules file for the entire host; deprecate and remove custom Mutex class in favor of std::mutex; expand helper routines for the TalkgroupRulesLookup class (backwards compatibility essentially with TalkgroupIdLookup); fix issue in network core that could cause deactivated TGs to try using talkgroup lookups when talkgroup lookups may be unavailable; * support code for future use; * fixup CMakeLists to fix build warnings for ARM 32-bit platform; * update detection for sendmsg and sendmmsg (handles some platforms that don't define sendmmsg); implement container classes for RTP framing; * update CMakeLists to handle new files appropriately; * initial implementation of FrameQueue to generate FNE RTP frames; * fix incorrect return; * implement a typedef for unique_ptr uint8_t arrays; fix issue where DMR, P25 and NXDN control classes were accepting BaseNetwork instead of Network; refactor how FrameQueue handles returning messages read from the network; refactor BaseNetwork, Network and FNENetwork to use FrameQueue instead of raw network writes; * fix issue with handling network state; fix issue handling global network enable flag; fix buffer pointer cleanup; * add option to CMake to selectively utilize the legacy (non-RTP) network protocol (embedded FNE does not support this and will be disabled for USE_LEGACY_NETWORK; cleanup some error messages when trying to initialize and read the configuration file; fix some minor issues in parsing the new talkgroup rules YAML file; complete conversion of BaseNetwork from raw socket to using the new FrameQueue class; refactor the multi-buffer UDPSocket write function to properly send multiple messages in batch down to the kernel driver (on Linux using sendmmsg); add flag to UDPSocket to indicate whether the socket is "open" or not; correct some RTP formatting issues with the extension header (the payload length is in 32-bit units); * remove remaining native WIN32/WIN64 type support code (it was unmaintained anyway); correct issues with network socket reading; cleanup some error messages;q * finally resolve lingering issue with frame queue having random socket issues; * refactor how BufferVector and sendmmsg UDPSocket write operates; refactor FrameQueue to properly queue messages; * properly queue packets to send in bursts to peers; add missing ACKs; add missing MSTPONG; * fix up formatting; remove unnecessary debug statements; * minor cleanups; * remove unused function; * add fields for function and sub-function; * describe network functions and subfunctions; * reorganize some code; add checking for packet sequence; * roll next sequence properly; * utilize typedef for std::unique_ptr<uint8_t[]>; * fix usage of __UNIQUE_BUFFER to __UNIQUE_UINT8_ARRAY; * clean up macro definitions; * implement sequence counting for the FNE conference bridge; * remove USE_LEGACY_NETWORK support; * use macro expansions for this instead of raw buffer lists; * implement actual transmission of new network func/sub-func behavior; * deprecate use of packet tags for determining operation, instead use RTP FNE header function and subfunction bytes; * fix issue with send talkgroups not setting flag to send talkgroups to peers; fix issue when compiling list of activated or deactivated TGs to send to peers; correct strangeness with the RTP sequence counting; fix talkgroup rule lookup on peer when activating or deactivating TGs; * update copyright dates; * complain if the RTP and FNE headers don't agree on stream ID; * for when you forget to update a comment so things actually make sense when you look at it later...; * remove extraneous check, this is really not necessary since FrameQueue does the size validation; * clarify the description of the "embedded FNE"; * merge changes from https://github.com/CVSoft/dvmhost to place spec specific limits on channel spacing; * refactor how network traffic is handled, we don't handle protocol specific stuff within BaseNetwork anymore and offload that to the specific protocol controller instead; implement support to receive peer traffic and repeat (still need to do master to peer); * convert fatal error about Rx frequency being below base frequency to a warning instead; * fix incorrect redefine of slotNo; * correct ret flag for readDMR, readP25 and readNXDN not being set to true by default (stupid me); correct FNE-mode payload offsets for diagnostic packets; * implement option to selectively enable/disable FNE mode verbose logging information; enhance FNE display of network diagnostic logs; refactor dmr::Control::processNetwork(); refactor RTP packet sequence counting; * fix incorrect variable used for out-of-sequence log message; * fix issues with login RTP stream; specially handle certain control opcodes with a different payload type; continued enhancements to sequence counting; * whoops readd accidentally removed block; * backout previous lastPeerId code; properly ensure RTP packet sequence is repeated for traffic calls; * transmit DVM LC_CALL_TERM TSDU at the end of call or if a call drops (hopefully); * don't reset the packet sequence for a RPTPING; * rename some class variables for clarity and prep for some future changes; * add support for priority/immediate Tx queues, these queues are intended to be used sparingly (and mostly for control data) to transmit priority messages [this is an experimental change and may break the build!]; * fix issue where connecting to a non-existent REST API endpoint would crash with an unhandled ASIO exception; * increase timeout delay for the modem by 4 seconds (the original 4 second delay would be too short for some longer operations); * use the immediate queue to prioritize control data in some situations [this is experimental and may break things!]; * reformat headers; * implement toString() functions on TSBKs, CSBKs and RCCHs for future use to replace fixed strings; cleanup DMRDefines slightly; * fix up and simplify RTP timestamp generation; * reduce reliance on fixed strings, instead rely on toString() functions from TSBK, CSBK, and RCCH to ensure string consistency for logging purpsoes; * refactor how and when releaseGrant is called in some situations; ensure channel grant messages traverse the network; * fix reset of error count between network super-frames; * add development helper file for VS Code to configure debug launch parameters; * update CMakeLists to be more condusive to debugging; * cleanup AUTHORS.md; * implement parrot for the conference bridge FNE; implement appropriate call detection logic for conference bridge FNE and attempt to prevent call collisons for the same destination; * add support for altering softpot levels from calibration mode; * bump copyright information; * reorganize CMakeLists; * remove finalcut from the libraries list (this isn't supposed to be included yet!); * implement initial rework of setup/calibration mode into a TUI; add ENABLE_TUI_SUPPORT CMake option to disable *all* project TUI support; add ENABLE_TUI_SETUP CMake option to disable setup TUI support for dvmhost; * properly update isConnected flag; * fix issue with incorrectly placed control; * part 1 of a fix for #30, the change here would be to allow the VCs to know about the CC to be able to report back to it to release or other wise update grants; * part 2 of the fix for #30, this implements the actual logic for a VC to notify the CC of a grant status change (touch, release) [NOTE: this could have undesirable consequences and this commit is experimental!]; * update README.md; * fix terrible copy and paste job resulting in a poor logic check (SMH); print to log the control channel address and port at startup; use the appropriate REST API for touch (whoops); * remove transmit overlay on shell; add F12 hotkey to trigger transmit; implement support to use F2 and F12 from various configuration/calibration modals; correct some mode strangeness when saving; * implement support to adjust FIFO buffer sizes from the host; * add engineering menu to adjust FIFO buffer sizes from setup; * ensure set button state is set properly for these windows; * fix stupid --tags non-sense in the git hash reporting; * ensure certain error conditions for setup mode are displayed when exiting at initial startup; * more fixes to attempt to satisfy #30 (why is this being so difficult!); correct situation where RESTClient may throw an assert instead of handling an error condition nicely; * for #30 allow release tg grant and touch tg grant to work regardless of dedicated CC setting; * file cleanups; * C++11/14 cleanups, class overrides and pure-virtual implementors /worked/ but were not really correct; * stringify modem command and reason messages (so log errors are easier on the eyes); * late update during a branch merge, to fix some latent issues that were missed;	3 years ago