Programming and Scripting :: MyDSL info parser

Also, sort of off-topic...

What is it that mydsl-load does initially that creates a pause before it begins to download? When I'm not connected to the internet it takes something like 10 seconds or so before wget starts. If I'm connected it's not nearly that bad, but still takes a couple of seconds.

I tried the sql version a little, although I don't yet understand how some of the features will be used. It was a little odd initially...I was offline and it created an empty database. After going online I had to manualy delete the database before it would download a new one.
____

Made more changes to the interface of the plain-text version. The file list can now be toggled with Alt+L. This was frustrating me for a long time because no matter what I did I couldn't get it to work if the list width was scaled smaller than its original width. I ended up creating it tiny and then resizing before showing the window. It looks like it might be a FLTK bug.

After running update or install, the filelist now gets back focus. This is to improve keyboard control.

The release date shown in the search results is formatted yyyy/mm/dd if the info originally is in mm/dd/yyyy or mm-dd-yyyy format. If the info has dd/mm/yyyy format, it will not convert properly...I don't think its possible to be 100% accurate in this case. I hope this will not be needed at some point in the future.

The "last update" box now also displays the total number of packages in the database.

I still want to find a way to grab the browser line that is associated witha specific info table index to make the code a little more streamlined, but so far my tests have caused more problems than they solve.

EDIT: added a function to jump from section to section in the browser, [ to go backward and ] to go forward

Code Sample

#!/home/dsl/bin/murgaLua-0.6.4
--[[
MyDSL info browser, 2008 mikshaw

Changelog (yyyy/mm/dd)
2008/03/17: More interface tweaks
Convert date format to yyyy/mm/dd (hopefully a temp fix)
2008/03/16: Interface tweaks
2008/03/15: Improved the update function
Database date check uses os.date/lfs instead of os.popen/ls/awk
Use wizard instead of tabs for better use of resources
2008/03/14: Undid gmatch change due to improvements to the database file
Improved keyboard control
Use Fl_Menu_Button for menus to add to keyboard control
Fixed crash on search or install with an empty database
2008/01/24: Fixed string.gmatch to break up files even if there is no newline between them
2008/01/17: Added date of last update
Tweaked the update function (it's still sloppy)
Fixed ".mydsl_dir not found" error message
2008/01/16: Fixed resizable behavior
Uses file in .mydsl_dir
Added auto update for first run
2008/01/15: Added text search
2008/01/14: Fixed display of incorrect duplicate info file
Check for trailing spaces in title string
Added "Update" function
Code and interface tweaks
2008/01/13: start of project
--]]

mydsldirfile=io.open("/opt/.mydsl_dir")
if not mydsldirfile then
print(arg[0]..": Can't open /opt/.mydsl_dir")
os.exit(1)
end
mydsl_dir=mydsldirfile:read("*l")
mydsldirfile:close()

listfile=mydsl_dir.."/mydslinfo.bz2"
listurl="ftp://ibiblio.org/pub/Linux/distributions/damnsmall/mydsl/mydslinfo.bz2"
list_sep="@b@B50@u"

function toggle(b)
local count,state,one,two=1
-- selected line and its label
local me,label=b:value(),b:text(b:value())
if string.find(label,"%+%s*") then
state,one,two="expanded","%+ ","%- "
else
state,one,two="collapsed","%- ","%+ "
end
b:text(me,string.gsub(label,one,two)) -- swap + and -
-- toggle all lines until reaching the next section or the browser end
while b:text(me+count) and not string.match(b:text(me+count),list_sep) do
if state=="collapsed" then b:hide(me+count)
else b:show(me+count)
end
count=count+1
end
end

function fill_list()
inputfile = io.popen("bunzip2 -c "..listfile)
data=inputfile:read("*a")
inputfile:close()
if not string.find(data,"%w") then -- if there's nothing readable in it
info_display_buffer:text("Cannot read database "..listfile.."\nPlease try updating again.")
else
filelist:clear()
header={}
info={} -- table containing all package data: info[index].{location,title,date}
info_count=0 -- number of packages
local current_loc="" -- location string
local headers=0 -- nuber of locations
-- breaks up info for each package into a separate string
for s in string.gmatch(data,"(Location:.-)\nLocation:") do
-- convert mm/dd/yyyy to yyyy/mm/dd
local datestring=string.match(s,"Current:%s*(.-%d)%s")
if string.find(datestring,"%d%d%D%d%d%D%d%d%d%d") then datestring=string.gsub(datestring,"(%d%d)%D(%d%d)%D(%d%d%d%d)","%3/%1/%2") end
info_count=info_count+1
info[info_count]={
location=string.match(s,"Location:%s*(.-)%s*\n"),
title=string.match(s,"Title:%s*(.-)%s*\n"),
date=string.gsub(datestring,"%D","/"),
--datestring=string.match(s,"Current:%s*(.-%d)%s"), -- plain
text=s
}
-- if not info[info_count].date then info[info_count].date="" end
if current_loc ~= info[info_count].location then
-- create a header when location string changes
current_loc=info[info_count].location
filelist:add(list_sep.."- "..current_loc)
headers=headers+1
header[headers]=filelist:size()
end
filelist:add(info[info_count].title)
end
for i,v in ipairs(header) do filelist:value(header[i]); toggle(filelist) end
last_update:label("last update "..os.date("%d %b %Y",lfs.attributes(listfile).modification)..", "..info_count.." packages")
end
filelist:value(1)
tab_browse:setonly()
end

function updatedb()
-- download to temp file, check file integrity, and only then replace the old file
tempfile=os.tmpname()
os.execute("aterm +tr -bg white -fg black -geometry 80x4 -title \"Database Update\" -e wget "..listurl.." -O "..tempfile)
if os.execute("bzip2 -t "..tempfile) == 0 then
os.rename(tempfile,listfile)
fill_list()
else fltk.fl_alert("Database check failed.\nThis may be the result of an incomplete download\nor gremlins in your internets.")
os.remove(tempfile)
end
filelist:take_focus() -- for better keyboard control
end

function pick_search_result()
-- show selected info file in the browse tab
if results:value() > 1 then
get_loc=string.match(results:text(results:value()),spacer.."(.*)"..spacer)
get_fname=string.match(results:text(results:value()),"(.-)"..spacer)
-- select the package in filelist
local myloc=1
while myloc<=filelist:size() do
if not string.find(filelist:text(myloc)," "..get_loc,1,1) then
myloc=myloc+1
else filelist:value(myloc)
break
end
end
-- open submenu if not open
if string.find(filelist:text(myloc),list_sep.."+ ",1,1) then toggle(filelist) end
while myloc<=filelist:size() do
if filelist:text(myloc) ~= get_fname then
myloc=myloc+1
else filelist:value(myloc)
break
end
end
for i=1,info_count do
-- make sure you get the right info AND category
if info[i].title == get_fname and info[i].location == get_loc then
info_display_buffer:text(info[i].text)
tab_browse:setonly()
myfile=info[i] -- this is the file that will be downloaded
break
end
end
tabs:value(tab1)
end
end

function list_cb()
if filelist:value()>0 then -- clicking a blank space can be fatal without this check
if Fl:event() == fltk.FL_RELEASE or Fl:event_key()==fltk.FL_Enter then -- on click or Enter
if string.find(filelist:text(filelist:value()),list_sep) then
toggle(filelist) -- if it's a separator
else
-- find out what submenu we're under
local myloc=filelist:value()
while myloc > 0 do
myloc=myloc-1
if string.find(filelist:text(myloc),list_sep) then
myloc=string.gsub(filelist:text(myloc),".-%s%s","")
break
end
end
for i=1,info_count do
-- make sure you get the right info AND category
if info[i].title == filelist:text(filelist:value()) and info[i].location == myloc then
info_display_buffer:text(info[i].text)
myfile=info[i]
tab_browse:setonly()
break
end
end
end
end
end
end

function toggle_list()
if listw > 0 then
oldw=listw
tiles:position(listw+5,nil,6,nil)
listw=0
info_display:take_focus()
else
tiles:position(6,nil,oldw+5,nil)
listw=oldw
filelist:take_focus()
end
end

ww=500; wh=360; bh=30; bw=ww/2-5
win=fltk:Fl_Window(ww,wh,"MyDSL Info Browser")

tabs=fltk:Fl_Wizard(0,0,ww,wh-bh-5)
tab1=fltk:Fl_Group(0,0,ww,wh-bh-5,"browse")

-- the tiles are built weird initially (filelist is only 1px wide)
-- this is a kluge; toggle_list doesn't seem to work if filelist
-- is shrunk less than it's initial width
tiles=fltk:Fl_Tile(5,5+bh,ww-10,wh-(15+bh*2))
--filelist=fltk:Fl_Hold_Browser(5,5+bh,ww/2-5,wh-(15+bh*2))
filelist=fltk:Fl_Hold_Browser(5,5+bh,1,wh-(15+bh*2))
filelist:callback(list_cb)
--info_display=fltk:Fl_Text_Display(ww/2,5+bh,ww/2-5,wh-(15+bh*2))
info_display=fltk:Fl_Text_Display(6,5+bh,ww-11,wh-(15+bh*2))
info_display:textfont(fltk.FL_SCREEN)
info_display:textsize(12)
info_display_buffer=fltk:Fl_Text_Buffer()
info_display:buffer(info_display_buffer)
fltk:Fl_End() --end tiles
tiles:position(6,nil,ww/3,nil) -- resize the filelist
tiles:callback(function() listw=filelist:w() end)
listw=filelist:w()
prevheader=fltk:Fl_Button(ww,wh,bh,bh,"&[")
prevheader:callback(
function()
for i=table.getn(header),1,-1 do
if header[i] < filelist:value() then
filelist:value(header[i])
break
end
end
filelist:take_focus()
end
)
nextheader=fltk:Fl_Button(ww,wh,bh,bh,"&]")
nextheader:callback(
function()
for i=1,table.getn(header) do
if header[i] > filelist:value() then
filelist:value(header[i])
break
end
end
filelist:take_focus()
end
)
-- offscreen button that allows the list to respond to Enter key
enter=fltk:Fl_Return_Button(ww,wh,10,10)
enter:callback(function() filelist:do_callback() end)
togglist=fltk:Fl_Button(ww,wh,10,10,"&l")
togglist:callback(toggle_list)

install=fltk:Fl_Button(ww/2,5,bw,bh,"&Install Selected Package")
install:callback(
function()
if myfile then
mydslload=os.execute("mydsl-load "..myfile.title.." "..myfile.location)
if mydslload~=0 then fltk.fl_alert(myfile.title..":\nDownload or Checksum error!") end
end
filelist:take_focus()
end
)

b_updatedb=fltk:Fl_Button(5,5,bw,bh,"&Update Database")
b_updatedb:callback(updatedb)
fltk:Fl_End() --end tab1

tab2=fltk:Fl_Group(0,0,ww,wh-bh-5,"search text")
search_text=fltk:Fl_Input(85,5,ww-90,bh,"search text ")
search_text:when(fltk.FL_WHEN_ENTER_KEY)
search_text:callback(
function()
if search_text:value() ~= "" and info_count and info_count>0 then
results:clear()
if Fl_Browser.column_widths then
results:add("@B49@b@uPackage Name"..spacer.."@B49@b@uLocation"..spacer.."@B49@b@uRelease Date")
else
-- default screen font does not have bold or underline
-- and no columns means only one format string is used
results:add("@B49PACKAGE NAME"..spacer.."LOCATION"..spacer.."RELEASE DATE")
end
for i=1,info_count do
if string.find(string.lower(info[i].text),string.lower(search_text:value()),1,true) then
results:add(info[i].title..spacer..info[i].location..spacer..info[i].date)
end
end
end
end
)

results=fltk:Fl_Hold_Browser(5,5+bh,ww-10,wh-(15+bh*2))
-- column widths available in 0.6.4+
if Fl_Browser.column_widths then
results:column_widths({results:w()/2,200,0})
spacer="\t"
else spacer="\t\t" -- force some extra space for older versions
results:textfont(fltk.FL_SCREEN) -- fixed-width makes it closer to columns
end
results:callback(
-- react to click only
function()
if results:value() > 1 and Fl:event()==fltk.FL_RELEASE then
pick_search_result() end
end
)
-- offscreen results_callback button for Enter key
results_cb=fltk:Fl_Return_Button(ww,wh,bh,bh)
results_cb:callback(pick_search_result)
fltk:Fl_End() --end tab2
fltk:Fl_End() --end tabs group

-- using buttons instead of tabs gives better keyboard control
tabbuttons=fltk:Fl_Pack(5,wh-bh,200,bh-5)
tabbuttons:type(fltk.FL_HORIZONTAL)
tabbuttons:spacing(5)
tab_browse=fltk:Fl_Radio_Button(0,0,100,bh-5,"&browse")
tab_browse:when(fltk.FL_WHEN_CHANGED)
tab_browse:box(fltk.FL_BORDER_BOX); tab_browse:selection_color(7)
tab_browse:callback(function() tabs:value(tab1) end)
tab_search=fltk:Fl_Radio_Button(0,0,100,bh-5,"text &search")
tab_search:when(fltk.FL_WHEN_CHANGED)
tab_search:box(fltk.FL_BORDER_BOX); tab_search:selection_color(7)
tab_search:callback(
function()
tabs:value(tab2)
Fl:focus(search_text)
end
)
fltk:Fl_End() --end tabs buttons
--last_update=fltk:Fl_Box(bw+5,wh-bh-5,bw,bh+5)
last_update=fltk:Fl_Box(ww-5,wh-bh-5,5,bh+5)
last_update:align(fltk.FL_ALIGN_LEFT)

win:resizable(tabs)
tabs:resizable(tab1)
tabs:resizable(tab2)
tab1:resizable(tiles)
tab2:resizable(results)
win:show()
find_file=io.open(listfile)
if find_file then data=find_file:read("*a") else data="" end
if string.find(data,"%w") then
find_file:close()
fill_list()
else updatedb()
end
Fl:run()

Quote

Interesting that you would think an initial offline status would work.
Not that I needed debugging there. Beside use of the File menu ->UpdateDB would fetch it when you are online. No need to manually delete it.

The flow of the GUI is to type an valid SQL statement and upon presssing Enter, a hit list is displayed. Selecting one from such list switches to the View tab displaying the fields select in the SQL statement. Whereupon pressing enter will install the extension or select SQL one can make another choice from the displayed hit list or even decide to further expand or refine the SQL statement.

The QBE will be used to automatcially build the SQL statement via entering the data via corresponding fields. Kinda like SQL "training wheels". Once again the hit list and the generated SQL will be shown and selecting one from the hit will display the view of all fields selected just as before.

Quote (mikshaw @ Mar. 16 2008,14:58)

Thanks for sharing. I'll try it very soon.

Quote

Oh, and this works with DSL 4.2.5 or DSL 3.4.11 no need to grab latest murgaLua

Are you implying there is trouble with my experiment beyond the lack of columns in murgaLua versions prior to 0.6.4? I tried to make sure it works in 0.5.5, but maybe I missed?

All of your scripts specify

Quote

#!/home/dsl/bin/murgaLua-0.6.4

Which would naturally lead one such assumption as it is not needed in DSL unless features that are only available in 0.6.4.were being attempted.

Quote (mikshaw @ Mar. 16 2008,15:22)

Many factors here including but not limited to:
Multiple entry points to the mydsl loading system to support CLI as well as GUI. A mix of many shells and lua scripts being called. And the internet host name lookup based on configuration file. All of this before the actual download begins.

Next Page...
original here.