Buurst® SoftNAS 5 Release Notes

Improvements:

• PHP - Upgraded PHP from version 7.4 to 8.2, enhancing performance and compatibility.
  • Security update - Sensitive parameter redaction protects sensitive data from leaking in stack traces by redacting those values.
• ZFS - Updated ZFS to version 2.2.3, improving stability and functionality.
  • Improved bug fixes: Resolved various bugs including data corruption issues and improved Zpool trim performance on Linux.
  • Foundation for future Linux kernel compatibility establishes the groundwork forfuture compatibility with the latest Linux kernel (6.8).
  • Other improvements:
    • Added support for Ntfy notifications for ZED (ZIL Encryption Daemon).
    • Various minor improvements across the codebase.
  • Timeout - Addressed timeouts experienced during updates, ensuring smoother operations.

Bug Fixes:

• AWS - Addressed a problem where the routing table failed to update after automatic failover in HA setups.
• Azure - Aligned Azure CLI with the latest SoftNAS version for improved compatibility.
• Notifications - Confirmed standard email notification cadence, resolving default setting discrepancy.
• Safari - Fixed Safari WARNING pop-up bug in Expansion Pool flow, allowing users to select disks/RAID levels correctly.
• Updates - Enhanced SoftNAS updates process stability for smoother experience.
• RHEL/Rocky - Fixed installation issues for RHEL/Rocky Linux versions.
• Remediated Qualys QIDs: 86729 and 38739 for improved security.
• SNAP Replication/HA
  • Improved security by introducing a more secure SSH key exchange between nodes.
  • Fixed fingerprint change issues by clearing and reacquiring existing SSH keys, ensuring smooth HA reactivation after updates.

Known Issues:

• RHEL- Updated RHEL Installer compatibility to accept versions 8.6 - 8.9, addressing installation issues.

All Identified CVEs

• Critical Vulnerabilities (CVEs) -We’ve addressed highly critical and/or critical vulnerabilities since the previous SoftNAS release. Rocky Linux security update for Kernel includes:
  • CVE-2022-38096)
  • (CVE-2023-6931)
  • (CVE-2023-6546,ZDI-CAN-20527)
  • (CVE-2024-0565)
  • (CVE-2023-51042)
  • (CVE-2021-33631)
  • (CVE-2024-1086)

Feature Enhancements

• Offline Activation - Users can now activate instances without internet constraints. Even in offline scenarios, users can set up instances seamlessly.
• Direct OCI Integration - Users can seamlessly access Oracle Cloud Infrastructure (OCI) resources directly from the VMware environment expanding storage options and flexibility.
• Enhanced Storage Pool Expansion Flow -User gains precise control over the storage infrastructure via a process flow to choose the correct disk size and type within the Storage Pool optimizing functionality and performance.
• ZFS Upgrade to Version 2.2.0 - User benefits from stronger data protection, smoother container integration, and granular control. 
• Smarter DeltaSync - User’s backups now trim unnecessary files, freeing up storage space on both local devices and cloud platforms.

Bug Fixes

• Instant UI Loading - The StorageCenter now loads instantly, for better user experience.
• Efficient UI Access via SSH Port Forwarding - User can now connect to the StorageCenter seamlessly, including when using SSH port forwarding.
• DNF Updates - Updates now run without hiccups, eliminating “dnf fail” messages.
• Stable File System Quotas during Updates - Quotas now remain consistent during updates, ensuring stability.
• Uninterrupted Winbind Daemon - The Winbind daemon on target nodes now runs smoothly.
• Improved Nginx Experience - Upgrades for Nginx are now obstacle-free, enhancing performance and security.

Known Issues

• Storage Pool Expansion Confirmation Prompt - When expanding Storage Pools, unchecking “Automatic Filtering” may hide the confirmation prompt. We are actively addressing this for smoother pool updates.

All Identified CVEs

• Critical Vulnerabilities (CVEs) - We’ve addressed highly critical and/or critical vulnerabilities since the previous SoftNAS release. These include:
  • CVE-2023-20900 (open-vm-tools)
  • RLSA-2023:7549 (Rocky Linux Security Update for Kernel)

• AWS - Added platform disk device designations to Add Disk menu. 
• Sticky Bit - Addressed security issue where sticky bit was not set on certain directories. 
• Kernel - Removed older, unused kernel packages from system to address issue where security scans were producing false positives. 
• All identified CVEs - Identified highly critical and/or critical vulnerability issued since the previous release of SoftNAS (RLSA- 2022:5826 - Rocky Enterprise Software Foundation Product Errata (rockylinux.org), RLSA-2023:3847 – Kernel version updated to 4.18.0-477.21.1). 

• AWS – Added support for IMDS v2. 
• Azure – Resolved issue where SSH key was not deployed properly to new virtual machine. 
• StorageCenter – Updated default RAID level to start at 0 for new pools. 
• Known Issue – Removed unresolvable entries from /etc/hosts file. 
• New Features
  • Added security enhancements for data in-transit.
  • Improved SnapReplicate implementation.

• Azure - Resolved SNAP HA Takeover fails due to change in underlying AZ CLI command.  

• AWS - Resolved AWS Account Secrets not masked when setting up SNAP HA. 

• Command Line - softnas-cmd cli now supports SoftNAS licensing. 

• StorageCenter – Fixed UI where an occasional redirection of user occurs which leads to Invalid License Condition Page. 

• License  
  • Fixed leading or trailing space in license key code which caused invalid license condition. 
  • Resolved invalid license condition encountered after setting up SNAP HA Marketplace billed image. 

• All identified CVEs - Identified highly critical and critical vulnerabilities issued since the previous release of SoftNAS (RLSA-2023:1566, RLSA-2023:1252, RLSA-2023:1405, RLSA-2023:1569, RLSA-2023:1930, RLSA-2023:2810, RLSA-2023:3246 & RLSA-2023:1572).  

• Known Issues  
  • Unstable machine ID could be created in the cloning of the SoftNAS instance. Prior to a user attempting to clone the instance they need to contact support@buurst.com.  
  • Users may encounter a “404 Not Found” or “invalid license condition” page when updating from 5.4.0 to 5.4.1. This redirection is due to an issue in the 5.4.0 release but does not impact the update process. The update will continue to run in the background and be completed successfully. It is recommended that the system is not rebooted or powered off when this is encountered. The update process will reboot the system once it is completed. 
  • Azure - SoftNAS is not supported under Azure “Virtual machine scale set”. Please see documentation here. 

• Monitoring Tool - Added new monitoring functionality including critical metrics such as real-time data, alerting functions & event logs for business decision making. For more information click here.  
• User Notifications - Added new notification features via StorageCenter console and email to inform users on performance including performance, expiry date, license type & instance entitlement. 
• Product Licensing – Improved efficiency of new license key generation & management. 
• All identified CVEs - Identified highly critical and critical vulnerabilities issued since the previous release of SoftNAS (CVE-2022-4269, CVE-2022-4378 & CVE-2022-4378).  
• Known Issues - During an in-place upgrade Storage Center UI hangs on Safari but the installation completes successfully.  

• All identified CVEs - Identified within previous versions of SoftNAS that were considered critical and high vulnerabilities including identifiers CVE-2022-2873, CVE-2022-41222 & CVE-2022-43945.

• VMware - Resolved default name server issue in OVA.

• SnapReplicate - Fixed issue with missing ssh_host_rsa_key.
• ZFS - Upgraded to 2.1.9.
• CIFS Shares - Select user popup not displaying configured users was resolved.
• SMB - Added option to enable or disable CIFS encryption to StorageCenter UI. Please see documentation found here.

• All identified CVEs - Identified within previous versions of SoftNAS that were considered critical and high vulnerabilities.

• Pool Creation - Fixed an issue when pool creation sporadically failed on AWS.

• HA configurations on Azure - Addressed issue with Azure storage accounts that could result in a "split brain" failure with HA configurations.
• SMTP - Fixed issues identified with user settings of the SMTP Encryption.
• Deltasync - Reduced the amount of storage needed for the logs. Fixed an issue where a Deltasync could potentially hang due to too many system files being open. 

• All identified CVEs - identified within previous versions of SoftNAS that were considered critical and high vulnerabilities

• All identified CVEs - identified within previous versions of SoftNAS that were considered critical and high vulnerabilities specific to Oracle, Rocky Linux 8, and PHP 7.4.x.
• StorageCenter UI - several reported issues including disallowing the expansion of storage pools with different RAID levels, types of disks and sizes.
• GP3 support on AWS - added support for creating and editing AWS GP3 disks.
• SNAP HA - increased deltasync performance.

Known Issues 5.2.6

Deltasync snapshot - when a deltasync operation is performed you may encounter an error event for the message "Remote Delta Snapshot created successfully". The error can be ignored and will be resolved in a future release.

• All identified CVEs - identified within previous versions of SoftNAS that were considered critical to be addressed including CVE-2021-4115.
• License Update - minor changes to license management scheme being used.

• All identified CVEs - identified within previous versions of SoftNAS that were considered critical to be addressed including CVE-2021-4034.
• Kernel  - upgraded to the latest version currently available which addressed specific issues identified (4.18.0-348.12.2).
• Zfs - upgraded to the latest version currently available which addressed specific issues identified (2.1.2).
• Btier  - upgraded to the latest version currently available which addressed specific issued identified (2.1.29).
• Increased support on Azure -  resolved issue where certain Azure Instance Types support a larger number of disks that was consuming all of the device names in the current supported format being used (i.e. /dev/sd[b-z]).  Resolved this by modifying the format to /dev/sd[a-z][b-z].
  

• Additional Monit Checks - pool capacity checks.
• DriftBot - Removed from product.
• Lift and Shift - provide a way to cancel automatic share/volume size calculation.
• Nginx - Updated version to resolved outstanding CVEs.
• Red Hat Enterprise Linux 8.4 - support added in response to v8.2 end of support.
• Rocky Linux 8.5 - moved to Rocky Linux due to End of Support for CentOS Linux.
• Slack and Microsoft Teams - added as options for automatic monitoring notification.
• Snapshot scheduling - enhanced scheduling user interface and added a safe guard to ensure users do not delete volumes when trying to delete snapshots.
• Softnas-cmd - added iSCSI specific commands along with adding a flag to "executeupdate" to be able to only update using minor version.
• StorageCenter Administrative Interface - enhanced overall look and feel of the user interface.

• CIFS - resolved issue where a share was not being properly restored from a pool backup.
• DeltaSync - resolved an issue where opened Microsoft Office documents were not being properly handled.
• Install/Update - addressed several areas of cleanup regarding some minor errors reported in the software update details.
• Lift and Shift - improved performance and resolved an issue where new flows using a Private CIFS Share failed.
• NFS - increased NFS thread count and NFSv3 firewall port management corrections.
• SNAP HA/SnapReplicate - had several issues addressed in this release including
  • resolved an issue where status was not being properly displayed when communication was lost between the nodes
  • corrected a calculation error that resulted in the progress of a replication to be reported incorrectly
  • resolved an issue where a double failover could occur due to a race condition
  • resolved an issue where it takes too long to reassign the VIP post failover. Now VIP and NAS-services reassignment is among the top list of the failover process
• StorageCenter Administrative Interface - resolved issue where the root disk of a VMware deployment was being displayed under Disk Devices, also resolved a couple of broken links within Quick Helps and Documentation.

• Install/Update - when performing an update using the StorageCenter Administrative Interface, the progress bar will seem to be "stuck" even though the update is still proceeding as expected.  If you are using this method to perform the update, expanding the Update Details pane will permit you to see the actual progress still being made. This is due to the Nginx update to this release. 

• Enhanced security by updating config file permissions.

• Disabled NFS ID mapping may impact ability to perform updates behind a firewall.

• Deployments using S3 or Azure Blob storage may not successfully update without configuring end points.

• Snapshot and Disk Activity may not reflect location time.

• VNET location may not be found when deploying new HA configuration.

• Downsizing of an AWS instance may result in boot failure.

• Samba may crash resulting in SoftNAS drives unmounting frequently.

• Nginx may not properly handle SSL certificates.

• SoftNAS NFS Thread Counts low due to nfsd not starting on new deployments.

• Identified “race condition” causing HA/SnapReplicate to fail manual takeover.

• Monit action blocking HA giveback.

Known Issues 5.1.3

Yum update - it has been observed occasionally when running a "yum update" from the command line you may experience an error message when the kernel debug package is installed.  This is contributed to the fact SoftNAS uses a custom kernel package.  The error can be ignored and will be resolved in a future release.

winbindd log message - There are some situations where RPC functions return RPC faults and this is not a fatal condition.  In that situation a dcesrv_call_dispatch_local: DCE/RPC fault in call lsarpc:32 - DCERPC_NCA_S_OP_RNG_ERROR message will be observed in the /var/log/messages file.

• Snapshot and Disc Activity may not reflect location time.

• VNET location may not be found when deploying new HA configuration.

• Downsizing of an AWS instance may result in boot failure.

• Samba may crash resulting in SoftNAS drives unmounting frequently.

• Nginx may not properly handle SSL certificates.

• SoftNAS NFS Thread Counts low due to nfsd not starting on new deployments.

• Identified “race condition” causing HA/SnapReplicate to fail manual takeover.

• Monit action blocking HA giveback.

• SnapHA prematurely starts before remote node is ready.

• Manually creating NFSv3 shares lack write privileges.

• Increased NFS thread count for improved performance.

• Security updates and fixes, notably CVE-2021-3177.

SmartTiers Pools/Volumes/SnapClones not remounted issue - An issue occurring in some SmartTiers deployments, where pools, volumes, and associated SnapClones were not mounting after a reboot has been resolved. 

HA installation failure issue - Resolved an issue where high availability (SNAP HA) would fail to install due to an assumption that the two VMs were in the same resource group. Logic added to ensure that subnet info is consulted prior to connection attempt. 

RHEL Root Corruption and Disk Unmountable issue - A known RHEL issue related to THP (transparent huge pages) default setting resulted in metadata expanding beyond memory capacity, resulting in data corruption. THP is now disabled at boot by default to prevent this issue re-occurring. 

Unable to create Azure Blob (object) Storage - An issue in which creation of blob storage on Azure would result in "unknown error" has been resolved. 

Invalid Help Link when connecting with SSH - Updated support link displayed when connecting by SSH to your SoftNAS instance/VM to ensure customers have easy access to support. 

Updated video links and content for in-app help - Updated or added video links to in-app help items, and added new help content to app features. 

Lift and Shift Azure Storage Type Selection issue - An issue in which the customer could not select disk type for Lift and Shift flows on Azure instances has been resolved. 

Failed takeover from UI - A race condition that could result from manual takeover/giveback operations that could result in takeover failure due to instance/VM roles switching places too quickly has been resolved. 

Failed error reporting issue - In some isolated cases, SnapReplication would not report failure if previous cycle completed successfully, and the next cycle was unable to start due to target node being down. This could result in replication showing as functional when it was not. This has now been resolved. 

Lift and Shift Home (Flexfiles) Repository available mappable target - An issue in which the home repository for Lift and Shift was listed as available as a target for mapping (Lift and Shift should not target the home directory) has been resolved.

Lift and Shift Failure to Create Repository - An issue in which attempts to create a Lift and Shift repository has been resolved. (19181)

SSL Certificate Issue between VMware and AWS Lift and Shift operations - Necessary communication between VMware and AWS instances for Lift and Shift operations were being blocked by certificate configurations. This has now been resolved. 

SnapReplicate Throttling issue - An issue where throttling command was not installed by default, resulting in potential poor performance, has now been resolved.

Nginx comment line issue - An issue where a SED REGEXP would constantly add # to the nginx registry file whether commented already or not, potentially resulting in large files sizes over time, has now been resolved. 

Nginx SSL certification corruption issue - An issue in which failover for HA would not work due to Nginx SSL Cert corruption has now been resolved. 

Custom NFS IDmapping configuration issue - An issue with a custom script in a client environment has been connected to an unsupported NFS setting. NFS IDmapping has been disabled in SoftNAS 5.1.1 for security reasons. Enabling NFS IDmapping is not a recommended configuration. 

• Added needed support for RHEL 8.2 and supporting installer.​

  • SoftNAS RHEL 8.2 BYOL Only​

  • Installed onto an existing fresh install of RHEL 8.2​

  • Installed via an installer script which will be downloaded from our website.  ​

The installer and documentation can be found at: RHEL SoftNAS Installation Steps

• Ability to enable FIPS​

• Added AWS SSM agent

Failover Issue when using custom SSL certifications - An issue in which custom SSL certifications would be over-written during a failover operation, resulting in loss of high availability, has been resolved. The custom SSL certifications will no longer be over-written. 

Error when modifying NFS shares from UI - An issue in which attempts to modify an NFS share from the Create an NFS Share section of the UI would result in failure and an error message have been resolved.

Security Fixes - The following fixes have been made to secure SoftNAS 5.1.0:

• removed vulnerability to CVE-2021-3156 (sudo related) 

• removed kernel vulnerabilities to CESA-2020:4286, CESA-2020:4431 and CESA-2020:4685 ​

• Netatalk completely removed from the system

Yum Update Issue - An issue preventing yum updates from completing has been resolved.

Benign Log Error - Log errors of a specific type seen occasionally on install has proven to be benign. The error reads  "Job for <service name> failed because the control process exited with error code"

Service names that have been seen for this error include:

• webmin.service

• php-fpm.service

• sernet-samba-smbd.service

• sernet-samba-winbindd.service 

These log errors will be addressed in a future release.

• Security Fixes - Recommended fixes have been applied to secure SoftNAS from recent Linux vulnerabilities, including CVE 2021-3156 and CVE 2020-9850.

• Yum Update Fix - Fixed an issue where yum update would fail in most recent version of SoftNAS, resulting in error messages related to selinux. 

• Azure Custom Data fix - Fixed an issue preventing the ability to use custom data on VM launch.

• Deprecated Function Fix - An issue in which deprecated features (FlexFiles and UltraFast UI) were listed as available. These features are now in our separate product, Fuusion. 

• DeltaSync Failed State issue - Resolved an issue where DeltaSync would fail to sync any new volumes added to an instance while HA was deactivated. 

• Notification Email issue - If the email notification was not configured it was logged as an ERROR, but is now configured to log as INFO, to prevent users from interpreting this as an actual ERROR in the SoftNAS logs.  

• Force Sync fix - Force sync now requires a volume selection to be made. This prevents the user from accidentally issuing a Force Sync on all volumes.

Build Packaging Fix - Fixed the build packaging for platform publishing.

License enforcement - As of 5.0, license enforcement will be applied to only allow SoftNAS servers to operate that are within the license entitlement performance level.

Linux version change - SoftNAS is now based on a CentOS 8.2 Linux deployment.

Kernel update - SoftNAS' kernel has been updated to version 4.18.0-193.19.1.el8_2.x86_64

Samba update - Samba has been updated to version 4.12-10-15

AWS EIP Support - SoftNAS no longer supports the use of Elastic IP addresses on versions 5.0 and later

Apple File Protocol (AFP) Support - AFP is no longer supported as of versions 5.0 and later

Upgrade process improvements - SoftNAS no longer checks and interrupts the upgrade process if the version of paired instances with SnapReplicate is different. The upgrade process will proceed without issue. 

Firewall default settings change -To improve the baseline security of SoftNAS, Firewalld default settings have been changed. All ports not required for use of SoftNAS will now be closed. If any services connect with SoftNAS through extraneous ports in your existing deployment, you will need to open these ports in SoftNAS's firewall.

Azure OMS Agent Support - Azure ASM and OMS Azure Extensions are now supported.

Improved performance - Performance improvements of up to 60% have been benchmarked for certain workflows with SoftNAS 5.0

SSH User change - With the update to the latest CentOS build, the centos user is now used for SSH connections to the virtual machine, rather than the ec2-user.

Snapshots are now time zone aware - Snapshots have now been changed to use UTC when naming snapshots and will display according to the local time zone regardless of the location and time zone of the snapshot datastore. 

Common User Configurable NTP settings - Fixed an issue in which some SoftNAS Services prioritized a hard-coded time server over user configured time server settings. 

Enable custom data for Azure - In order to allow automation of deployments and allow images to self-configure and bootstrap with predetermined values, SoftNAS now supports custom data.

Azure L2ARC Boot Hang issue - Fixed an issue where Azure instances without ephemeral drives could hang for up to 15 minutes during boot. This would typically manifest itself with the inability to login to to the instance via the UI or SSH. 

CLI License Activation Issue - An issue in which license activation via the CLI would not function has been resolved. 

Unable to edit snapshot schedule via CLI - An issue in which edits to snapshot schedule using the CLI would not be recognized as valid, resulting in a null value, has been resolved. 

Max Threads Cleared issue - An issue in which :SoftNAS.ini" script would not retain the max threads count, necessary for Lift and Shift setup, has been resolved.  

Azure Storage Expansion issue -  An issue in which  storage expansion operations on  Azure virtual machines with attached read/write logs (L2ARC or ZIL) would fail has been resolved.

Inconsistent HA State issue - Fixed an issue where the displayed HA state between two SoftNAS nodes could be inconsistent.  

SmartTiers Resize issue - Fixed an issue where the UI could hang if resizing a SmartTiers tier. Occasionally this issue could result in an invalid license alert.  

Lift and Shift Fixes

Lift and Shift EFS Source selection issue - An issue in which the EFS source could not be selected from the UI in order to be mounted (but could be mounted from the CLI) has been resolved.  

Miscellaneous Fixes

Update related bucket error - An issue in which after an update would occasionally result in an InvalidBucketName error when using CEPH based storage has been resolved.  If using CEPH based object buckets please contact support prior to the upgrade for assistance.

Notification Email issue - If the email notification is not configured it is logged as an ERROR, but should display as an INFO status in logs. This will be resolved in an upcoming release. 

NiFi Monit Alerts without NiFi enabled - NiFi Monit alerts may appear in your logs even if NiFi has not been enabled on your SoftNAS VM. 
Recommended Action: Disable NiFi Monit Alerts in the Service Monitoring section under Settings>Administrator>Monitor Settings.

BYOL License Email not sent - Email alerts for SoftNAS BYOL performance level violations are not sent for Azure VMs. BYOL performance level violations are detected and displayed in the UI.
Recommended Action: The running VM needs to be down-sized to be compliant with the installed license or a license appropriate for the VM in use needs to be applied.

SnapReplicate/SNAP HA un-synched volume issue - A volume or volumes added while an HA pairing is deactivated will not replicate to the secondary node upon reactivation of HA. 
Recommended Action:  Run Force Sync for the unsynchronized volumes before re-enabling SnapReplicate/SNAP HA. Ensure the volumes are selected prior to issuing the Force Sync. If no volumes are selected a Force Sync will resync all volumes. 

CIFS/SMB import issue - When importing an S3 device with CIFS/SMB enabled to a new instance, the existing CIFS/SMB volume(s) are not automatically imported to the new instance.
Recommended Action: Open the Pools section of the Storage Administrator panel, and perform an import of the pool. 

yum update issue - yum updates on version 5.01 are erroring out with no action taken. This will be resolved in an upcoming release.