The December 2022 Edition of The Resource
Hello Reader, here is this month’s iRODS news and developments!
If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’.
News
Happy Holidays!
Here’s hoping I get this out in time for Christmas. As I write this, the snow is heavy on the ground, a rare thing in East Anglia these days!
Festive greetings to you all.
TriRods
The December talk was from Martin Schobben, and featured a demo of rirods, the first release of a pure R package for iRODS, and discuss its design and implementation.
Martin’s a friendly and knowledgeable fellow I had the pleasure of meeting at the UGM this year, so if you want to or are using R with iRODS, please do check the talk out.
iRODS Development Update: November 2022
iRODS Globus Connector 4.2.11.2 is released
No 4.3.x yet, it seems (or Release/Change logs :grumpyface:).
Changes between 4.2.11.1 and 4.2.11.2
and addresses, as far as I can make out;
- The update time was not being updated due to the thread which closes the file not actually writing data.
- created own base64 encode/decode to resolve conflicts
How’s the upgrade from 4.2.7 to 4.2.11 going?
Last month I was struggling to get all the dependencies lined up (partly, it turns out, some hard coded assumptions in the Ansible play book I was using). Having got past that, I’ve run into some (separate) issues around Federation, Audit and Tiering plugins. The issues aren’t public facing yet, because at least the Federation and Tiering ones are hard to report without leaking a lot of internal site data. It’s also really hard to build a reproducer - the Zones go back twelve years and multiple versions, and it could be anything related to those, or nothing!
Main Repository Activity
Open Issues
Using std::memcpy() on a l1desc object can lead to undefined behavior
Allow usage of client-side and server-side filesystem library implementations in the same translation unit
General queries cannot use attributes than may be NULL in where clauses
ils should not segmentation fault when listing with a ticket
read_metadata access level not behaving as advertised
4.3.0
What you're reporting is correct.
We have only exposed the permission levels in between the original 4 levels.
remove imeta rmw (and addw?) (and lsw?)
4.3.0
That is correct. No message is printed to the terminal, but the help text does mention deprecation.
See #6417
Error for password too long only occurs for passwords between 43 and 45 characters.
iput cannot create a new object with a ticket
iRODS support for SQLite
It doesn’t have this at the moment, but an interesting thought for test deployments, so one doesn’t have to spin up an MySQL or PostgreSQL for each test system/container.
Investigate if unrecoverable signal handlers should be set in main server process
Investigate updating CRON task configurations on configuration reload
These are Delay Server CRON, not cron(5) / Unix cron!
DBus error
I am not sure if you would call this a feature or bug report. We have a kubernetes cluster that does not allow systemd for processes. Is it possible to start iRODS without systemd?
Sounds like this is still under investigation.
Deprecate and replace all use of uint
Deprecate unused member variables in l1desc_t
Calls to boost::condition_variable::wait can result in spurious wakeups
Investigate authentication plugins self-reporting scheme name
How does irods-grid interact with tree voting and resources marked down
If you use irods-grid and passthru write/read weighting and iadmin modresc XXX status down or any combination, I’d recommend you read this, as there are some nuances that it will help to be aware of.
Feature request: allow irods-grid to provide ips information
Boo! It got bumped from 4.3.1 to 4.3.2! I was hoping it was a ‘quick fix’ (is there every such a thing?) :-)
iRODS should not allow collection and data object to have same name if they are in same parent collection
One of our users has created a data object and a collection with the same name, both of which belong to the same parent collection. From what I can tell, the user created a collection named Week5 Lab in the collection Bio352. Next they uploaded a data object named Week5Lab into Bio352. Finally, they renamed the collection Week5 Lab to Week5Lab. I think they were using Jargon 4.2. I cannot reproduce the issue using iCommands.
extractVarNames can result in a buffer overflow
client_connection.hpp causes duplicate definitions of irods::experimental::defer_connection
iqstat -a crashes - cannot handle wrong input
Duplicate of #6267
( iqstat -a should return INCOMPATIBLE_PARAMS targetted for 4.2.12)
Delay rule execution fails with MySQL in 4.2.11
Failure to install irods-auth-plugin-krb on Ubuntu 18.04
We have not released the kerberos plugin for 4.3.0 as we intend for kerberos requirements to be met by the new pam_interactive plugin.
You can install iRODS 4.2.11 if you need kerberos functionality today.
Using std::exit in signal handlers is not async-signal-safe
Use sigaction for signal handling
Closed Issues
Closed on - 2022-12-15 21:12:45 Restore python function used by rsZoneReport API
rsZoneReport fails when generating the zone report because it attempts to invoke the following python function which was removed during the transition to Python 3.
Closed on - 2022-12-14 22:14:56 Can remove an object with my delete_metadata access level
Closed on - 2022-12-14 19:50:47 modify_metadata access let user remove metadata
Closed on - 2022-12-14 22:14:41 create_metadata access level includes more access than its scope
Closed on - 2022-12-07 14:01:41 C client fails to build against iRODS 4.3.0 release
Baton users take note!
For now, this can probably be worked around by avoiding including irods/rodsClient.h in baton's header files. Any structs needed for function declarations can be forward declared.
It’s all ready for 4.3.1 though, due to #6448.
I was able to build baton against the irods-{dev,runtime} built from the main branch. That fix will be included in 4.3.1.
Closed on - 2022-12-01 17:13:26 Add clang-format and clang-tidy github workflows to C++ projects
Closed on - 2022-12-05 15:03:10 Add .clang-format file to C++ projects
Closed on - 2022-11-24 19:05:29 iqstat should output details of any ruleId once run by a rodsadmin
I hope I am not missing anything here. As an rodsadmin user I can list all available rules in delay queues by iqstat -a. As normal I can also list details of a specific rule that belongs to me by iqstat ruleId. Furthermore, I can list the details of a ruleId that belongs to a user that I know by iqstat -u user1 ruleId. However I couldnot see the details of available rules that I could list without knowing the user accounts that made those rules.
Closed because
iqstat -la lists all.
Closed on - 2022-12-14 01:00:26 icommands error message unclear on non-existent directory
What did you try to do?
run iget -f to retrieve a file to a non -existent dir
Yes, that is weird, but not a bug.
That has to do with the OS and reference counting. See https://stackoverflow.com/questions/12359598/delete-current-directory-on-linux
Closed on - 2022-11-22 19:28:57 Incorrect use of gethostname() and HOST_NAME_MAX
fixed for 4.2.12.
I think I am reading this as irods expects 64 byes, but gethostname allows 255.
Closed on - 2022-12-15 21:11:38 Rename base64_encode and base64_decode
Closed on - 2022-11-22 19:29:05 irods::client_api_allowlist::enforce is marked noexcept, but can throw exceptions
Closed on - 2022-11-21 21:19:56 Refactor user administration API to throw exceptions instead of return error codes
Closed on - 2022-11-21 21:16:19 irodsDelayServer does not start because FQDN is configured as delay_server leader
Fixed in the son to be released 4.3.1
The implementation for delay server migration only considers the name returned by gethostname.
Closed on - 2022-12-13 22:41:05 rstrcpy needs to log source string on error
Closed on - 2022-11-21 15:40:25 Document detached mode for unixfilesystem resources
Exiting!
Closed on - 2022-12-15 22:29:37 test_parallel_transfer_engine on branch with apparently unrelated changes.
Closed on - 2022-12-14 15:28:38 using a too long servername should not stacktrace
Closed on - 2022-12-12 21:41:57 Inconsistent ichmod behavior with inheritance of strict ACLs and data_owner_name
Fixed in 4.3.1
Yes, this is due to ichmod honoring the data_owner_name. It should not. This will be fixed alongside #6579.
Python iRODS Client Activity
Open Issues
raise error in testing if server is more recent than advertised IRODS_VERSION
investigate deprecation of open()’s use/honoring of kw.DEST_RESC_NAME_KW
open() currently will honor the kw.DEST_RESC_NAME_KW keyword.
kw.RESC_NAME_KW and kw.RESC_HIER_STR_KW are also honored by open() - so it may be that there is no reason to also support kw.DEST_RESC_NAME_KW as it is designed for use with replication (with a pairing of source and destination).
Executing logical_quotas_get_collection_status with irods_rule_engine_plugin-logical_quotas-instance prints nothing
create a utility module for miscellaneous functions
Closed Issues
Closed on - 2022-11-29 20:09:12 Group based permission works differently for collections
The keyword argument report_raw_acls controlls whether individual users in a group not exposed by default. This behaviour is now modified;
Maybe we make a session.acls method that is "nearly synonymous" except it behaves more consistently than session.permissions (ie the default for the report_raw_acls parameter would be True). and at some future point, we'd deprecate session.permissions.
Closed on - 2022-11-29 20:08:58 Getting permission does not bring g: prefix for groups
Querying acl on (getting permission of ) an object doesn't bring g prefix specifying group based permission although there is already an granted group permission exists. AccessManager behaves it like a normal user. On the other hand ils -A shows it is a group user.
which is now fixed by
#411 adds a user_type attribute to the iRODSAccess.
This will have the value "rodsgroup" in the analogous situation where ils -A would print a "g:" prefix.
Closed on - 2022-12-12 13:24:38 Uploading data raises a KeyError -1205000 once the logical quota starts monitoring
After a lot of back and forth analysis on both sides;
Created an issue in the Logical Quotas plugin repo , including how to reproduce this same error -1205000, with the only plugin installed or active being logical quotas:
irods/irods_rule_engine_plugin_logical_quotas#84
In that issue, istream is used rather than PRC to evoke the error, but the same effect can be achieved in PRC with this put script and a single argument specifying a very small file to upload:
So if you use Logical Quotas plugin, it’s worth checking out rodsuser cannot upload to own collection with quota in force #84
Closed on - 2022-12-14 21:48:29 PRC presents itself to servers as a 4.3.0 compatible client
Good clients must report the server version they are maximally compatible with else they run the risk of using old behavior (e.g. bad packstruct/xml encoding - apos vs graves).
I had a look through iRODS Docs and irods_library_examples, and I couldn’t find this written down, but I am no C coder, so perhaps it’s obvious if you write from that angle. If not, then you might want to note the above.
NFSRODS Activity
Closed Issues
Closed on - 2022-12-07 18:27:19 write access, singularity (possible misconfiguration)
I think that this is saying that the latest NFSRODS version is compatible with >= 4.2.9, and so the reporter using 4.2.8 wouldn’t have worked.
This is in the README for 2.1.0 of the plugin, and became true in 2.0.0. Unfortunately, there isn’t a table of what version works with which, so you have to track this down yourself.
icommands Activity
Nothing this month
externals Activity
Nothing this month.
YODA Activity
Open Issues
[FEATURE] Don’t show all possible email addresses known in Yoda instance
When adding a new user to a research-group, one can see all kinds of user email addresses available in Yoda including personal email addresses. These addresses have not been included by the user themselves. The image below is from my test account in Yoda DGK at UU. The same is also true for Yoda@WUR which shows a much longer list of users when you for example search on gmail to add a user. Users have indicated that this is undesirable to be able to see private email addresses of others. I'm not sure how this relates to the GDPR either, I can imagine the system does not require showing email addresses to the user to function correctly (and hence data limitation is not applied?).
[FEATURE] customisation of data classification terms selector
At WUR we have a different data classification terms in our security policy
It would be nice to be able to have customised set of data classification terms for use during group creation and metadata editing.
which turns out to already be possible with theme packaging, however do be aware that;
We will lose this modification if the default schema is upgraded, so it might be a good idea to not put the enumNames in metadata.json itself (the same goes for the default value of affiliation).
which I think is the reason this issue is still open.
Closed Issues
Closed on - 2022-12-14 11:27:57 [BUG] DataCite metadata contains an incorrect date field.
See DOI 10.48338/vu01-2lt5v4
The metadata contains a Date field with "Date Type" "Updated". This was set to 2022-12-08Y11:57.46.000000.
When you try to update the dataset in DataCite Fabrica you can't update the DOI and it shows an error message stating:
"To save this DOI, first resolve the errors with these properties: date."
This field is not in the Yoda metadata so is probably automatically set in the Yoda to DataCite conversion.
Removing the decimals 2022-12-08Y11:57.46 resolves the issue.
Closed on - 2022-12-14 15:52:38 [FEATURE] Value of DataCite Publisher field should be configurable
When we publish a dataset the value of the Publisher field in the DataCite metadata is set to "Utrecht University". In our case this should be "Vrije Universiteit Amsterdam"
A quick search suggests the Publisher value is set in https://github.com/UtrechtUniversity/yoda-ruleset/blob/release-1.8/json_datacite41.py lines 153-155
This can't be changed at all by us and is difficult to change for SURF.
Closed on - 2022-12-14 13:39:34 [BUG] changing the subcategory in group properties for a datamanager group does not work
(I’ve slightly tweaked the text to include the error rather than a screenshot)
When trying to changing the subcategory of an existing datamanger group, I get the error
"Datamanager grup names may only contain lowercase letters (a-z) and hypens (-)
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
One Yak was shaved in the making of this newsletter, but it got a nice warm coat from Santa, so it’s not cold.