[Cosmo-dev] Bad ICS already extant

[Bobby Rullo]

I ran into the problem of Cosmo accepting "bad" ics, but then not  
being able to parse it when it came time to pull the EventStamp out  
of the database.

My issues was that ical4j doesn't like date properties without time  
that don't have "VALUE=DATE", which is ironic considering it was  
ical4j itself that created the ics in question! Apparently this IS a  
requirement in the spec, but according to our resident icalendar guru  
(a Mr. J Harris) there are many clients which emit data like this out  
there - iCal being one of them (not for DTSTART, DTEND, but for  
Alarms apparently)

It's conceivable that this was the problem in some (most? all? none?)  
of the migration issues we saw yesterday. If so, we might want to be  
patch ical4j to be liberal in accepting this data somehow, maybe even  
rewriting it as it comes in.

I like Randy's fix (although concerned about performance) because it  
it fails loudly when you store the data, not some time in the future  
when you forgot what you did. So people attempt to put data into  
cosmo using their favorite client, it fails, and we fix Cosmo to  
accept the data, if it's reasonable (reasonable meaning the client  
has enough users we care about, the client is not doing something  
completely absurd, etc.)

Bobby

On Feb 1, 2007, at 9:34 AM, Randy Letness wrote:

> Jared Rhine wrote:
>> Randy's r3540 says:
>>
>> * Perform additional validation on Calendar object by
>> validating serlialized Calendar can be parsed again. We
>> ran into a problem where bad .ics was allowed into db,
>> messing everything up. There is an inconsistency in
>> ical4j's validate() and parsing (validate() says its ok
>> but parsing barfs.) The idea is to never allow bad .ics
>> into the db.
>>
>> What can we do about bad data already in the database?  This seems  
>> to affect the data migration process.  These ICS resources seem to  
>> have gotten into and out of 0.5 ok, as these real-world calendars  
>> have been in daily use, so something during 0.6 development got  
>> more restrictive?
>>
>
> The change in 0.6 is that once the initial .ics was stored,  
> subsequent updates using the internal Cosmo api's didn't do a full  
> parse, only a Calendar.validate().  It turns out ical4j's  
> Calendar.validate() was returning true for something that the  
> CalendarParser didn't like.  This meant bad .ics was getting into  
> the db.  I added the additional parse check to prevent this, but it  
> shouldn't effect 0.5 data as that data has already run through parse 
> ().
>
>> (I'm a little concerned about interoperability, as there is bad  
>> ICS out there in the real-world, no doubt.  Now, there will be  
>> some resources that Cosmo can't store; instead an error will be  
>> returned, right?  Which error, btw; will there be a way to  
>> quantify the number of resources being rejected from say the logs?)
>>
>
> Yes, a 400 Bad calendar objetct error should be returned.
>
>
>> How does this interact with Chandler's behavior during publishing?  
>> Presumably Chandler allowed this data and submitted it during  
>> publication, and may do so again.  Will publication of collections  
>> containing such resources fail or just some events not succeed?
> It depends, if you are talking about the old Chandler publish than  
> if Chandler submits bad .ics then it will fail.  If you are talking  
> about morse code, then this doesn't apply because there is no .ics  
> being transmitted.
>
> -Randy
> _______________________________________________
> cosmo-dev mailing list
> cosmo-dev at lists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-dev