Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

Call For Proposals of User Group Meeting 2022!

​Register here​

iRODS Internship - Summer 2022

​5 C++ Projects​

Is that more than last year? I feel it is…

iRODS Development Update: March 2022

​March 2022​

iRODS S3 Resource Plugin v4.2.11.2 is released

​S3 plugin - change list ( No changelog… Yet?).

iRODS Client: Metalnx v2.6.0 is released

​Metalnx - change list - and changelog - new for this version? Hurrah!

Python iRODS Client v1.1.3 is released

​Python iRODS Client - change list and Change Log.

Nice small release!

Main Repository Activity

Open Issues

​Update processes around unattended installations​

for 4.3; “The following tasks need attention (not exhaustive):

Update JSON schema for unattended_installation.json
host_access_control_config -> host_access_control
hosts_config -> host_resolution
Update documentation in docs.irods.org
Update setup_irods.py"

​Delay server should not log a stacktrace when default config value is used​

​Delay server adds completed rules back to queue, race condition, then complains loudly​

Currently seen on main inly - not sure if that also includes 42-stable and 43-stable, though…

​Centos 7 requires pyodbc externals package​

"Before 4.3, iRODS contained a complete copy of pypyodbc. It was a single file that lived in this repo."

And so far…

"After some discussion, we've decided that this will be handled via documentation for 4.3.0.
The plan is to provide a proper package for 4.3.1."

So you need to have the package installed when you upgrade to 4.3, ATM…

​Some methods of Test_msiDataObjRepl_checksum_keywords failing​

​CMake: include headers in target SOURCES + conditionally use file sets​

​Investigate PostgreSQL ODBC options: UseDeclareFetch and Fetch​

"Handling a very large database query resultset can lead to out-of-memory errors. This situation has been seen with PostgreSQL. One suggestion is to use the ODBC options UseDeclareFetch and Fetch to limit the number of results."

​Packaged python files with python3-only syntax cause packaging to fail on some Fedora-based systems​

​Add option that limits the number of rules the delay server is allowed to fetch from the database​

"Today, the delay server will fetch as many rows from R_RULE_EXEC that match the search criteria. This can lead to memory issues with the ODBC driver if there's a large amount of delay rules in the database.
To avoid this problem, we should provide an option that limits the number of rows found by the delay server (on wake up). This option would rely on SQL clauses such as LIMIT and TOP to accomplish this."

​Consider moving delay server server_config.json settings into R_GRID_CONFIGURATION​

​Ensure namelinks are only in irods-dev package​

​server-side irods_environment.json doc and validation schema bugs​

Specifically for irods-grid.

​Consider defining and adding constants for common database column sizes to Packing instructions​

Related to Delay Server work.

​Federated parallel transfer with SSL enabled fails with SYS_COPY_LEN_ERR​

Hoping this is just 4.2.11, as it will block our migration from 4.2.7…

​Client API Allowlist option does not align with description at docs.irods.org​

"https://docs.irods.org/4.2.11/system_overview/configuration/ states that the default value is enforce. This means, even when the property does not exist in server_config.json, it should be treated as if it were explicitly set to enforce.
The code does not follow this."

​CPack fails to byte-compile python3 syntax on Centos 7​

Main branch only.

​Wrap-around behaviour in configuration integers​

​Consider renaming functions dealing with shared memory in rodsServer.cpp​

​core.re, core.dvm, and core.fnm aren’t updated when iRODS upgraded​

TL;DR -“the core.re.template file will get updated by the packaging/installation.”

​Rule engine plugin framework fall through causes spurious failures, crashes, log spam​

A few takeaways from this issue;

"According to the 4.2.8 docs, delay rules support targeting a specific REP via the 

and

"In regards to your config, normally the cpp_default_policy REP is the last plugin in the list so that unhandled PEPs are caught and processed without falling through. But you have it in the middle. Is that on purpose?"

I don’t think there is a ‘best practice’ guide for using/implementing rule plugins, I wonder if we could crowdsource one from those who use it a lot? I’ve recently started and find it a steep learning curve!

​irepl -U fails to update stale zero byte replicas​

"Despite its presence in the documentation, the -U flag is no longer meaningful in 4.2.9+."

​Verify non-package installations work for 4.3.0​

Good to see this isn’t being forgotten, and a really nice write up - but then I am s sucker for a checklist!

​Remove absolute paths from config files and source code​

Should be helpful for Conda deployments, I would imagine…

​Delay rule execution frequency directive should work together with different units or comma?​

A fascinating discussion on the best practices around Execution Frequency ( in the rule language), along with some python code. I think this sort of think should be captured in a ‘best practice’ document - perhaps something that could be crowdsourced by the community?

​Ensure icommands user space tarball packaging still works for 4.3​

​Update get_script_output_single_line() to use Boost.Process​

​Consider replacing the use of vfork() with fork()​

​scoped_client_identity is not federation friendly​

"The library needs to update the zone in addition to the username.
This also applies to functions that adjust the clientUser inside of the RsComm."

​iqstat -a should return INCOMPATIBLE_PARAMS​

Aka ‘you’re holding it wrong’! One cannot ask for both all the rules and more detail on a single specific rule…

​iqstat should be able to target other zones​

"Currently iqstat, iqdel and iqmod don't have an -z flag to allow the user to talk to a Federated Zone.
Could they grow one please, to be similar to other ixcommands e.g. imeta?"

With the very reasonable response of;

"Okay, so, for now, maybe iqstat grows ability to list delay rules from 'over there'.
The manipulation of delay rules (iqdel and iqmod) might warrant more investigation/changes - separate issue."

​When composable resources voting and individual resource status conflicts, users don’t get their data​

This is fixed in >= 4.2.9, but for those of us running behind;

"What did you try to do?
iget a file where one replica out of two is stored on a resource that is marked down but the composable tree is configured with a higher read priority for that branch of the tree.
local resource branch (with marked down resource): resc_context: write=1;read=1
remote resource branch (with accessible replica): write=0.1;read=0.1
Expected behavior
The replica from the marked up resource is provided.
Observed behavior (including steps to reproduce, if applicable)
The voting logic from the passthru over-rides the resource marked down and the iget attempts to retrieve the replica from that resource, resulting in a HIERACHY_ERROR.
The only way for the user to retrieve that file is to directly target the replica with -n which requires a level of understanding of the state and make up of the underlying infrastructure that is not fair to expect of an end user (how do they know the other replicas isn't also unavailable in some way)."

​CMake: use CMAKE_PROJECT_HOMEPAGE_URL​

​CMake: use find_package for spdlog​

​CMake: use find_package for cppzmq​

​CMake: use find_package for zmq​

​CMake: use find_package for nanodbc​

​CMake: use find_package for avro​

​CMake: use find_package for fmt​

​CMake: use find_package for libarchive​

​CMake: use find_package for boost​

​CMake: find_package for ODBC​

​CMake: Find module for PAM​

Closed Issues

Closed on - 2022-04-14 18:49:23 Server should not start if JSON schema validation cannot be performed​

Closed on - 2022-04-14 18:48:52 Add static assertions for default delay server config properties​

Closed on - 2022-04-13 11:31:18 setup error - IrodsError: post-install test failed.​

Closed on - 2022-04-12 01:16:40 Add missing header files to irods-dev package​

Closed on - 2022-04-06 12:27:06 Remove rsLog.hpp and rsLog.cpp​

Closed on - 2022-04-06 12:26:13 Use human-readable timestamp instead of epoch seconds for filename suffixes​

Closed on - 2022-04-06 12:26:36 Remove unused JSON template files​

Closed on - 2022-04-06 12:26:53 Remove default_log_rotation_in_days from server​

Closed on - 2022-04-06 12:25:39 Add all missing server configuration options on upgrade​

Closed on - 2022-04-12 01:28:12 question about configuration of LDAP through pam​

Closed on - 2022-04-04 15:18:45 Consider moving the irods-grid icommand into the server​

"'Should we still include irods-grid in the icommands tarball package?'
I think no. Doesn't use the iRODS protocol. Should only come with the server (now that we've separated these).
'should we move it to /usr/sbin'
Feels correct. Cleaner separation, agreed."

Closed on - 2022-03-28 16:22:17 icommands and database plugin Debian packages should “provide” themselves like irods-runtime does.​

Closed on - 2022-03-30 16:10:46 setup of iRODS with existing non-local service account​

I think this also affects LDAP accounts, but I’ve been able to install against non-local accounts on 4.2.7, so perhaps this affects between 4.2.8 and 4.2.11?

Closed on - 2022-03-20 17:45:15 CMake: use POSITION_INDEPENDENT_CODE properties​

Closed on - 2022-03-20 17:43:43 CMake: Redundant definitions of BOOST_SYSTEM_NO_DEPRECATED​

Closed on - 2022-04-01 17:02:39 Latest jsonschema version seems unhappy with iRODS JSON schemas​

4.3.0 and jsonschema 4.0.0, perhaps earlier (later than 2.6.0)

Closed on - 2022-03-20 17:42:55 CMake: Use CMAKE_CURRENT_SOURCE_DIR and CMAKE_CURRENT_BINARY_DIR where appropriate​

Closed on - 2022-03-20 19:06:13 CMake: Do not add libm to CMAKE_CXX_STANDARD_LIBRARIES​

Closed on - 2022-03-20 18:19:10 CMake: use targets instead of variables for curl​

Closed on - 2022-03-20 18:22:26 catch2 externals package has include files in the wrong place.​

Closed on - 2022-03-20 18:21:50 CMake: Test compiler flags before adding them.​

Closed on - 2022-03-20 18:21:53 CMake: Use targets instead of variables for OpenSSL​

Closed on - 2022-03-20 18:20:12 Remove server dependency on irods-grid​

Closed on - 2022-03-20 19:05:03 irodsctl needs icommands to be in the PATH environment variable​

Closed on - 2022-03-20 18:58:59 Improve how the build system locates dependencies​

Closed on - 2022-03-21 16:54:44 msiTarFileExtract (or ibun) attempts to create directories on wrong server ​

Closed on - 2022-03-25 17:12:27 Mark queued delay rules as in-flight​

Closed on - 2022-03-26 00:02:37 platform.linux_distribution is deprecated in python 3.7 and removed in 3.8​

Closed on - 2022-03-21 18:50:47 Test failure (master only): test_all_rules.Test_AllRules.test_rulemsiServerMonPerf_r​

Closed on - 2022-03-21 17:30:19 Remote microservice does not error correctly with invalid hostname​

Closed on - 2022-04-15 15:36:06 Validate environment during ingest​

Closed on - 2022-04-05 16:13:59 Deprecate ASSERT_ERROR and ASSERT_PASS​

Closed on - 2022-04-06 00:41:56 server fails to start in rr intermittently​

Closed on - 2022-03-30 14:24:34 Python logging show timezone in timestamp​

Closed on - 2022-03-21 19:38:16 Change CPACK_PACKAGE_CONTACT and CPACK_PACKAGE_VENDOR​

Closed on - 2022-04-01 14:44:27 Rename irodsReServer to irodsDelayServer​

Worth casting your eye over before the upgrade to 4.3 if you have code that interacts with the Delay server, and/or you care about the number of concurrent processes it can run.

Closed on - 2022-03-25 23:35:07 signalExit/cleanupAndExit should not contain rodsLog calls​

Closed on - 2022-03-25 23:20:04 ichksum does not recognize invalid iCAT file length​

Closed on - 2022-04-14 18:47:58 adding ‘schema_name’ to all schema files​

Closed on - 2022-04-10 01:56:50 Enable use of “-“ in Zone names​

Can’t be fixed until V5(!) due to protocol changes.

"This is a legacy federation limitation.
iRODS 3.x used a hyphen in the RemoteZoneSID (tempZone-qwerty123). I'm not sure if the 4.x code is actually assuming that there is no hyphen. The validator is enforcing the 3.x assumptions.
The easiest thing in the near term (aka for testing) will be to use a local copy of the schemas via deploy_schemas_locally.py:
https://github.com/irods/irods_schema_configuration"

So as 4.3 uses a new schema, if your Zones contain dashes, you’ll have to modify the schemas and use a local copy, if I read this right…

Closed on - 2022-04-15 13:28:38 putDirUtil does not handle absolute symlinks correctly​

Closed on - 2022-04-15 16:23:55 imv to filename with slash does not return an error​

Closed on - 2016-08-12 20:18:23 add “schema_name” to configuration file schemas​

Python iRODS Client Activity

Open Issues

​fcntl not available on Windows​

" The fcntl is only for detecting the blocking/non-blocking behavior of a socket, but that is only for debug logging. We can easily opt out on the fcntl call under Windows."

Closed Issues

Closed on - 2022-04-07 13:17:07 Remove call to getpeername()​

"In irods/message/_init.py:_recv_message_in_len() we call socket's getpeername() to get host and port info for debugging purposes. Per Python socket documentation (https://docs.python.org/3/library/socket.html#socket.socket.getpeername), on some systems this function is not supported. Since this call was added to support debugging in case of a problem, I am removing it so the code works on any system. PR to be created shortly."

YODA Activity

Closed Issues

Closed on - 2022-03-31 07:03:22 [BUG] the second time a new user is added to a group, the link “Click here to add a new user to this group” disappears​

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

Some Yaks were shaved in the making of this newsletter.